Virtual Scenario Construction Method, Device, Equipment and Medium for Surgery

Abstract

An embodiment of the present invention discloses a method, device, equipment and medium for constructing a virtual scene applied to surgery. The method includes: determining scene construction information associated with a target user, and determining a target matrix to be processed of the scene construction information; wherein the scene construction basic information includes target basic information corresponding to the target user, at least one organ information to be processed associated with a target lesion, and target historical diagnosis and treatment information associated with the target lesion; processing the target matrix to be processed based on a pre-trained virtual scene construction model to determine a virtual surgery scene corresponding to the target user, so as to perform surgery simulation based on the virtual surgery scene. The technical solution of the embodiment of the present invention realizes the automatic and efficient construction of a virtual surgery scene, makes the generated virtual surgery scene more diversified, and also better meets the actual needs in the clinical scene.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a method, device, equipment and medium for constructing a virtual scene for surgery. Background Art

[0002] Currently, during the surgical training of medical staff, mechanical devices such as springs and gears are usually used to simulate human organs. At the same time, training materials are combined with three-dimensional images or augmented reality technology for display to train the clinical judgment ability and clinical processing ability during surgery.

[0003] However, there are still some defects in the solutions provided by the prior art. On the one hand, the process steps of surgical training are relatively single, and the training content is limited by the above mechanical devices and the initial settings of the images, and diverse surgical scenarios cannot be simulated. On the other hand, the surgical scenarios constructed in the above manner cannot fit the patient's condition and cannot meet the actual surgical training requirements in the clinical scenario. Summary of the Invention

[0004] The present invention provides a method, device, equipment and medium for constructing a virtual scene for surgery, which realizes the automated and efficient construction of a virtual surgical scene, makes the generated virtual surgical scene more diverse, and also more meets the actual needs in the clinical scenario.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a virtual scene for surgery, the method including:

[0006] Determine the scene construction information associated with the target user, and determine the target matrix to be processed of the scene construction information; wherein, the scene construction basic information includes the target basic information corresponding to the target user, at least one organ information to be processed associated with the target lesion, and the target historical diagnosis and treatment information associated with the target lesion;

[0007] Process the target matrix to be processed based on a pre-trained virtual scene construction model, and determine a virtual surgical scene corresponding to the target user, so as to perform surgical simulation based on the virtual surgical scene.

[0008] In a second aspect, an embodiment of the present invention further provides a device for constructing a virtual scene for surgery, the device including:

[0009] A scene construction information determination module, configured to determine the scene construction information associated with the target user, and determine the target matrix to be processed of the scene construction information; wherein, the scene construction basic information includes the target basic information corresponding to the target user, at least one organ information to be processed associated with the target lesion, and the target historical diagnosis and treatment information associated with the target lesion;

[0010] A virtual surgery scenario determination module, configured to process the target matrix to be processed based on a pre-trained virtual scenario construction model, determine a virtual surgery scenario corresponding to the target user, and perform surgery simulation based on the virtual surgery scenario.

[0011] In a third aspect, an embodiment of the present invention further provides an electronic device, which includes:

[0012] One or more processors;

[0013] A storage device for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual scenario construction method for surgery according to any one of the embodiments of the present invention.

[0015] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the virtual scenario construction method for surgery according to any one of the embodiments of the present invention when executed by a computer processor.

[0016] The technical solution of the embodiment of the present invention first determines the scenario construction information associated with the target user, and determines the target matrix to be processed of the scenario construction information, so as to determine the target basic information of the target user, at least one organ information to be processed of the target lesion, and the historical diagnosis and treatment information of the target lesion; further, based on the pre-trained virtual scenario construction model, the target matrix to be processed is processed to determine the virtual surgery scenario corresponding to the target user, and surgery simulation is performed based on the virtual surgery scenario, realizing the automated and efficient construction of the virtual surgery scenario. At the same time, by integrating information of multiple dimensions in the process of scenario construction, the limitations of traditional mechanical devices and initial imaging parameters are eliminated, making the generated virtual surgery scenarios more diverse and more in line with the actual needs in the clinical scenario. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the introduced drawings are only the drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of a virtual scenario construction method for surgery provided by Embodiment 1 of the present invention;

[0019] Figure 2 It is a schematic flowchart of a virtual scene construction method for surgery provided in the second embodiment of the present invention;

[0020] Figure 3 It is a flowchart of a virtual scene construction method for surgery provided in the third embodiment of the present invention;

[0021] Figure 4 It is a structural block diagram of a virtual scene construction device for surgery provided in the fourth embodiment of the present invention;

[0022] Figure 5 It is a schematic structural diagram of an electronic device provided in the fifth embodiment of the present invention. Detailed implementation manners

[0023] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. In addition, it should be noted that for the sake of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0024] Embodiment 1

[0025] Figure 1 It is a schematic flowchart of a virtual scene construction method for surgery provided in the first embodiment of the present invention. This embodiment is applicable to the situation of constructing diverse virtual surgery scenes based on multi-dimensional information. This method can be executed by a virtual scene construction device for surgery, and this device can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal, a PC terminal, or a server, etc.

[0026] As Figure 1 shown, the method specifically includes the following steps:

[0027] S110. Determine the scene construction information associated with the target user, and determine the target matrix to be processed of the scene construction information.

[0028] In this embodiment, the target user can be a user who needs to perform surgery on a certain lesion on the body. In order to construct diverse virtual surgery scenes that meet the actual clinical needs around the target user, first, it is necessary to obtain information in multiple dimensions as the data basis, that is, obtain the scene construction information associated with the target user.

[0029] Among them, the basic information for scenario construction includes the target basic information corresponding to the target user, at least one organ information to be processed associated with the target lesion, and the target historical diagnosis and treatment information associated with the target lesion. Specifically, the target basic information can be information such as the name, gender, age, etc. corresponding to the target user stored in the medical information database; at the same time, in order to determine the surgical object in the virtual surgery scenario, after determining the target lesion of the target user, it is also necessary to retrieve at least one organ information to be processed associated with the target lesion. It can be understood that the organ information to be processed not only includes the information of the organ corresponding to the target lesion, but also includes the information of the organ associated with this organ and possibly affected by the target lesion. For example, when the organ information corresponding to the target lesion is a liver angiogram image, the organ information to be processed also includes the angiogram image of the heart; further, in order to construct a corresponding virtual surgery scenario around the target lesion, it is also necessary to obtain the information of similar cases in the information database of the medical system, that is, the target historical diagnosis and treatment information. Exemplarily, when the target lesion is the liver, the surgical scenario information for the liver can be retrieved from a specific server and used as the target historical diagnosis and treatment information for reference when constructing the virtual surgery scenario.

[0030] In this embodiment, since these information need to be processed by the neural network model in the subsequent process, after obtaining the scenario construction information for the target user, data reconstruction and transformation are also required, that is, to determine the target matrix to be processed corresponding to the scenario construction information. It can be understood that the target matrix to be processed at least includes multiple elements corresponding to the target basic information, the organ information to be processed, and the target historical diagnosis and treatment information.

[0031] Those skilled in the art should understand that in the actual application process, a conversion function can be used to perform data loop calculation on the above-mentioned large amount of multi-dimensional data, and then generate a high-dimensional matrix corresponding to the scenario construction information and meeting the input requirements of the neural network model. The embodiments of the present disclosure will not be elaborated here.

[0032] S120. Process the target matrix to be processed based on the pre-trained virtual scenario construction model, and determine the virtual surgery scenario corresponding to the target user, so as to perform surgical simulation based on the virtual surgery scenario.

[0033] In this embodiment, the virtual scene construction model can be a pre-trained neural network model, whose input is the target matrix to be processed corresponding to the scene construction information, and the output is the data basis required for constructing the virtual surgical scene. Specifically, the virtual surgery construction model can be pre-stored in the surgical training device. For example, it can be stored in the memory of a surgical training device that provides three-dimensional virtual reality and standard surgical procedures. Correspondingly, the determined virtual surgical scene can be displayed on the display of the surgical training device. Those skilled in the art should understand that the constructed virtual surgical scene can be a simulation environment of the real world, a semi-simulated and semi-fictional virtual environment, or a purely fictional virtual environment. In the actual application process, regardless of which of the above types the virtual surgical scene is, the specific information reflected by the scene is associated with the target lesion of the target user.

[0034] It should be noted that during the process of training the virtual scene construction model, it is necessary to first select historical scene construction information and the corresponding historical virtual surgical scenes from the historical scene database as the training set to train the model. Further, a part is selected from the historical scene database as the validation set to estimate the relevant parameters of the model. Finally, the performance of the model is evaluated and optimized using the test set, and the trained virtual scene construction model can be obtained.

[0035] Further, after the virtual surgical scene is displayed on the display of the surgical training device, the user can perform surgical simulations based on other components provided by the surgical training device. The simulation process includes clinical treatment operation training and clinical judgment training, etc. Continuing with the above example, when the virtual surgical scene of the target user's liver is constructed, the physicians participating in the surgical training can use the operation components on the surgical training device to perform simulation operations on the liver in the virtual surgical scene to rehearse the future surgery of the target user and predict the surgical results.

[0036] The technical solution of this embodiment first determines the scene construction information associated with the target user, and determines the target matrix to be processed of the scene construction information, so as to determine the target basic information of the target user, at least one information of the target organ to be processed of the target lesion, and the historical diagnosis and treatment information of the target lesion; further, based on the pre-trained virtual scene construction model, the target matrix to be processed is processed to determine the virtual surgical scene corresponding to the target user, so as to perform surgical simulations based on the virtual surgical scene, realizing the automated and efficient construction of the virtual surgical scene. At the same time, by integrating information in multiple dimensions during the scene construction process, it gets rid of the limitations of traditional mechanical devices and initial imaging parameters, making the generated virtual surgical scene more diverse and more in line with the actual needs in the clinical scenario.

[0037] Embodiment 2

[0038] Figure 2 This is a schematic flowchart of a virtual scene construction method applied to surgery provided in the second embodiment of the present invention. On the basis of the foregoing embodiment, after determining the multi-dimensional scene construction information of the target user, the target taboo information can also be determined to simulate intraoperative emergencies; use a pre-set template to screen redundant information in the scene construction information, realizing templatized sorting of information; perform dimensionality reduction processing on the target matrix to be processed based on the node similarity sub-model, and input the obtained feature sequence to be processed into a pre-trained GAT model, facilitating the surgical training system to construct a virtual surgical scene based on the high-dimensional matrix; provide feedback on the virtual surgical result based on the real-time retrieved surgical evaluation knowledge graph, improving the intelligence of the surgical training system. The specific implementation manner can refer to the technical solution of this embodiment. Among them, the same or corresponding technical terms as those in the above embodiment will not be elaborated here.

[0039] As Figure 2 shown, the method specifically includes the following steps:

[0040] S210. Determine the scene construction information associated with the target user, and determine the target matrix to be processed of the scene construction information.

[0041] In the process of determining the scene construction information associated with the target user, since information in different dimensions differs in data type, data format, and data storage and transmission methods, the corresponding determination methods are also different. Optionally, the method for determining the target basic information is to obtain the to-be-processed basic parameters corresponding to the target user, and extract the target basic information corresponding to the preset fields from the to-be-processed basic parameters; the method for determining the to-be-processed organ information is to determine at least one to-be-processed organ information from the to-be-processed views associated with the target lesion based on an image recognition algorithm; the method for determining the target historical diagnosis and treatment information is to determine the target historical diagnosis and treatment information associated with the target lesion in the historical diagnosis and treatment information database according to the pre-established diagnosis and treatment knowledge graph.

[0042] Specifically, the to-be-processed basic parameters corresponding to the target user include the patient's basic information, such as age information, gender information, family information, etc., and may also include the patient's physiological index information, such as specific data of the patient's body temperature, heart rate, and blood pressure, and may also include the preoperative tracking information of the patient, such as the electrocardiogram information and blood pressure change curve of the patient within a specific time period. It can be understood that the above information can be obtained through the corresponding information acquisition module. Further, after obtaining the to-be-processed basic parameters containing the above information, the part required for constructing the virtual scene can be extracted therefrom according to the pre-set field extraction rules as the target basic information, thereby realizing the elimination of redundant data.

[0043] In this embodiment, the organ information to be processed at least includes the contrast screening information of the target lesion and the similar case information. Exemplarily, when it is determined that the target lesion of the target user is the liver, the scanned or contrast image of the patient's liver can be used as the organ information to be processed. Similar to the physiological index information and the preoperative tracking information, the scanned image or the contrast image can also be extracted from the corresponding scanning device or imaging device. At the same time, from the image database of the medical system, the information of other patients who have undergone liver surgery can be screened out, and then these information can be sorted out to construct a corresponding set as the similar case information.

[0044] In this embodiment, in order to determine the construction basis of the virtual surgery scenario, it is also necessary to determine the target historical diagnosis and treatment information associated with the target lesion in the historical diagnosis and treatment information database based on the diagnosis and treatment knowledge graph. Specifically, the target historical diagnosis and treatment information can be determined in the target historical information database based on the diagnosis and treatment knowledge graph after extracting information from the contrast screening information and the similar case information.

[0045] Among them, the knowledge graph is a modern theory that combines the theories and methods of disciplines such as mathematics, graphics, information visualization technology, and information science with methods such as bibliometric citation analysis and co-occurrence analysis, and uses a visual graph to vividly display the core structure, development history, frontier fields, and overall knowledge architecture of the discipline to achieve the integration of multiple disciplines. Based on this, it can be understood that the diagnosis and treatment knowledge graph refers to a mapping map constructed by visualizing clinical diagnosis and treatment knowledge, that is, a structured graph showing the development process and structural relationship of diagnosis and treatment knowledge. As the carrier of historical diagnosis and treatment information, the diagnosis and treatment knowledge graph can at least be used to describe, mine, and analyze the correlation between the information involved in multiple historical surgery scenarios.

[0046] In this embodiment, the target basic information further includes the target taboo information corresponding to the target user. In the process of determining the target basic information, optionally, according to the pre-established taboo knowledge graph, the target taboo information corresponding to the target basic information is determined, and the target taboo information is updated to the target basic information to simulate intraoperative emergencies according to the target taboo information.

[0047] Among them, the target taboo information is the operation that needs to be avoided and the drug information that cannot be selected during the treatment of the target lesion. Similar to the diagnosis and treatment knowledge graph, the corresponding taboo knowledge graph can also be used when determining the target taboo information of the target lesion. Further, when the corresponding target taboo information is determined for the patient's target lesion, the information needs to be updated to the target basic information to use it as the data basis for constructing the virtual surgery scenario.

[0048] Exemplarily, when it is determined that the target lesion is the liver and there is a metabolic disorder in the patient's relevant organs, based on the taboo knowledge graph, some drugs prohibited during the operation can be determined, as well as the maximum dosage of anesthetic drugs for this patient. Further, the information of prohibited drugs and the maximum dosage of anesthetic drugs are updated to the target basic information of this patient, so as to be displayed on the display screen associated with the surgical training device after the virtual surgical scenario is constructed.

[0049] It should be particularly noted that after updating the target taboo information to the target basic information, this information can also be used to simulate emergencies during the virtual surgical training process, making the virtual surgical scenario more in line with the actual situation of the patient and enhancing the intelligence of the virtual scenario construction scheme.

[0050] In this embodiment, after determining the scenario construction information of the target user, in order to obtain the input of the model, it is also necessary to determine the target matrix to be processed corresponding to the scenario construction information. Optionally, the target basic information, at least one organ information to be processed, and the target historical diagnosis and treatment information are spliced to construct the target matrix to be processed.

[0051] Specifically, after determining the target basic information, at least one organ information to be processed, and the target historical diagnosis and treatment information, since their specific content involves macroscopic information (such as the basic information of the patient and similar medical records) and microscopic information (such as the angiographic images and physiological index information of the patient and similar medical records) in multiple dimensions of the patient and similar medical records, the redundant information in the scenario construction information can be screened using a pre-set template, so as to realize the templatized arrangement of the scenario construction information.

[0052] Further, after finishing the arrangement of the above information, in order to facilitate the input of data into the neural network algorithm model, the multi-dimensional scenario construction information can be first packaged into triples or multi-tuples that conform to the Resource Description Framework (RDF). Among them, RDF is a data model represented using XML syntax, which is at least used to describe the characteristics of Web resources and the association relationships between resources. It can be understood that RDF provides a general framework for expressing data that can be processed by relevant application programs and enables it to be exchanged between application programs without losing semantics. Those skilled in the art should understand that for multi-dimensional scenario construction information, an RDF parser can be used to package the data, which will not be elaborated in this embodiment of the present disclosure. After packaging the above multi-dimensional scenario construction information, the corresponding target matrix to be processed can be obtained.

[0053] S220. Based on the node similarity sub-model, perform dimensionality reduction processing on the target matrix to be processed to obtain the corresponding feature sequence to be processed.

[0054] In this embodiment, the virtual scene construction model includes a node similarity sub-model and a Graph Attention Network (GAT) sub-model. It can be understood that before using the GAT model to process data and construct the corresponding virtual surgery scene, it is first necessary to use the node similarity sub-model to perform dimensionality reduction processing on the target matrix to be processed.

[0055] In this embodiment, before introducing the node similarity sub-model, the graph embedding process involved in the solution can be described first. When the data includes a high-dimensional matrix, problems such as slow input and operation may occur during the process of inputting the data into the corresponding machine learning model. Graph embedding is a process of mapping graph data (usually a high-dimensional dense matrix, such as the target matrix to be processed corresponding to the scene construction information in this embodiment) into a low-dimensional dense vector. Through the graph embedding process, the problem that graph data is difficult to efficiently input into machine learning algorithms can be solved. In this embodiment, by using the node similarity sub-model to perform dimensionality reduction processing on the target matrix to be processed, the data corresponding to the scene construction information can complete the above graph embedding process, which is convenient for the subsequent GAT model to process the data.

[0056] Traditional graph embedding methods such as Deepwalk, Line, SDNE, etc. are all based on the assumption of neighbor similarity, that is, the more common neighbors two vertices have, the more similar the two vertices are. However, in many scenarios, two vertices that are not neighbors may also have a high degree of similarity. For example, in this embodiment, the target user and similar cases are both associated with multiple dimensions of information, and the roles of these nodes in the neighborhood are similar. Therefore, in the actual application process, the struc2vec model can be selected as the node similarity sub-model to perform the above graph embedding process.

[0057] Specifically, struc2vec is a model proposed for capturing the structural role proximity of nodes. In the process of using this sub-model to perform dimensionality reduction processing on the target matrix to be processed, first, according to the neighbor information at different distances, the structural similarity of node pairs is calculated respectively, and then a multi-layer weighted directed graph is constructed, where each layer is a weighted undirected graph, and only the layers are directed; further, random walks are performed in the multi-layer weighted directed graph to construct a context sequence in the target matrix to be processed, and finally, based on the skip-gram training sequence, the low-dimensional vector representation of each node is obtained.

[0058] In the processed feature sequence obtained by dimensionality reduction of the target matrix to be processed, the content reflected by each feature sequence corresponds to the content of each dimension in the scenario construction information. Specifically, the processed feature sequence corresponding to the basic information of the target user can be P[b] = {a, s, f,...}, where a represents the age feature of the target user, s represents the gender feature of the target user, and f represents the family information feature of the target user; the processed feature sequence corresponding to the physiological indicators of the target user can be P[x] = {x1, x2,...}; the processed feature sequence corresponding to the pre-operative tracking information of the target user can be P[tx] = ∑P[x]; the processed feature sequence corresponding to the target taboo information determined based on the determined target basic information and the taboo knowledge graph can be D = f(∑P[b], ∑P[tx]); for at least one organ information to be processed, the processed feature sequence corresponding to the contrast screening information can be P[c] = {c1, c2,...}; the processed feature sequence corresponding to the similar case information can be P[r] = {r1, r2,...}; since the similar surgery information is determined from the historical diagnosis and treatment information database based on the contrast screening information and the similar case information, therefore, the processed feature sequence corresponding to the similar surgery information can be O = f(∑P[c], ∑P[r]).

[0059] S230. Process the processed feature sequence based on the pre-trained graph attention network sub-model to obtain a target feature sequence, and construct a virtual surgery scenario according to the target feature sequence.

[0060] In this embodiment, after dimensionality reduction of the target matrix to be processed to obtain the corresponding processed feature sequence, the graph attention network sub-model can be used to process it, and then the target feature sequence can be obtained. Optionally, input the processed feature sequence into the pre-trained graph attention network sub-model to obtain the attention coefficients of each element in the processed feature sequence respectively; determine the target feature sequence according to each attention coefficient and the corresponding element.

[0061] Since graph data usually contains the relationship between vertices and edges, and at the same time, each vertex also has its own features (i.e., the processed feature sequence), when processing data based on the traditional GCN model, it is not only impossible to process dynamic graphs, but also not convenient to assign different learning weights to different neighborhoods during the processing. Therefore, in this embodiment, the GAT model is used to process the processed feature sequence. During the processing, each vertex can perform the operation of attention coefficients on any vertex on the graph, and it is completely independent of the graph structure. It can be understood that, compared with other models, the GAT model does not need to use complex matrix operations such as eigenvalue decomposition. The following describes the processing process based on the GAT model.

[0062] When using the GAT model to process the feature sequence to be processed, it is first necessary to calculate the similarity coefficient e between each neighbor node and the vertex for the vertex ij , specifically, use a linear mapping with shared parameters to increase the dimension of the features of the vertex. For example, increase the dimension of the feature sequence to be processed corresponding to the basic information of the target user. Further, map the concatenated high-dimensional features to a real number to obtain the similarity coefficient e ij .

[0063] After determining the similarity coefficient between the neighbor node and the vertex, a softmax function can be used to calculate the attention coefficient a ij . Among them, softmax, as a function in machine learning, especially deep learning, can map some inputs to real numbers between 0 and 1, and the normalization ensures that the sum is 1. Finally, according to the calculated attention coefficient, the features are weighted and summed to determine the target feature sequence corresponding to the feature sequence to be processed. Exemplarily, when the feature sequence to be processed is , correspondingly, the target feature sequence output by the GAT model is

[0064] It should be noted that before processing the target matrix to be processed based on the virtual scene construction model, the model also needs to be trained. Optionally, obtain a training sample set; use the historical scene construction information in each training sample as the input of the virtual scene construction model to be trained, and use the actual surgical scene information as the output of the virtual scene construction model to be trained to train the virtual scene construction model to be trained; use the loss convergence in the virtual scene construction model to be trained as the training goal, and train to obtain the virtual scene construction model

[0065] Among them, the training sample set includes multiple training samples, and each training sample includes: historical scenario construction information associated with a historical user and actual surgical scenario information. The loss function is preset, and based on this loss function, the parameters in the virtual scenario construction model to be trained can be corrected. It can be understood that after training the virtual scenario construction model to be trained using the historical scenario construction information and actual surgical scenario information in the training set, the training error of the loss function, that is, the loss parameter, can be used as the condition for detecting whether the loss function has reached convergence. For example, whether the training error is less than a preset error, or whether the error change trend tends to be stable, or whether the current number of iterations is equal to a preset number. If the convergence condition is detected, such as the training error of the loss function reaches less than the preset error or the error change tends to be stable, it indicates that the training of the virtual scenario construction model to be trained is completed, and at this time, the iterative training can be stopped. If it is detected that the current convergence condition is not reached, the historical scenario construction information in other training sets and the corresponding actual surgical scenario information can be further obtained to train the model until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the trained model can be used as the virtual scenario construction model. That is, at this time, after inputting the target matrix to be processed associated with the scenario construction information into the trained virtual scenario construction model, the corresponding target feature sequence can be obtained.

[0066] In this embodiment, after determining the target feature sequence corresponding to the scenario construction information, the surgical training system can construct a virtual surgical scenario corresponding to the target user based on this data. Among them, the virtual surgical scenario includes virtual user information, virtual organ information, virtual diagnosis and treatment information, and intraoperative emergencies. It can be understood that in the virtual surgical scenario, the basic information, physiological index information, and virtual image information of the organs associated with the target lesion can be simulated around the target user, and it also includes surgical equipment required for the virtual surgery. At the same time, the constructed virtual surgical scenario can be multiple scenario segments, and in each virtual surgical scenario segment, the unexpected situations during the surgery can also be simulated based on the target taboo information. Exemplarily, when the target lesion in the virtual surgical scenario is the liver of the target user, unexpected situations such as bleeding of other associated organs or a significant decrease in the virtual blood pressure value of the user may occur in some scenario segments.

[0067] Meanwhile, since the constructed virtual surgical scene can be composed of multiple scene segments, based on this, when medical staff, as training participants, conduct surgical training within this virtual surgical scene, the surgical training device can adaptively adjust the surgical scene in subsequent other scene segments according to the operation information received within this scene segment, thereby making the constructed virtual surgical scene more conform to the actual surgical situation, which is conducive to training the clinical operation handling ability and clinical judgment ability of medical staff. Further, when the training participants complete the surgical training in the virtual surgical scene, the surgical training system can also provide feedback on the virtual surgical result based on the operation information and the surgically evaluated knowledge graph retrieved in real time, improving the architecture of the virtual surgical training and enhancing the intelligence of the surgical training system.

[0068] In the technical solution of this embodiment, after determining the multi-dimensional scene construction information of the target user, the target taboo information can also be determined to simulate intraoperative emergencies; the redundant information in the scene construction information is screened using a pre-set template, realizing the templatized organization of information; the target matrix to be processed is dimensionally reduced based on the node similarity sub-model, and the obtained sequence of features to be processed is input into a pre-trained GAT model, facilitating the surgical training system to construct a virtual surgical scene based on the high-dimensional matrix; the virtual surgical result is fed back based on the surgically evaluated knowledge graph retrieved in real time, enhancing the intelligence of the surgical training system.

[0069] Embodiment III

[0070] As an optional embodiment of the above embodiment, Figure 3 is a flowchart of a method for constructing a virtual scene applied to surgery provided by Embodiment III of the present invention. To clearly introduce the technical solution of this embodiment, it can be described by taking the application scenario where a diverse virtual surgical scene is constructed based on multi-dimensional information as an example, but it is not limited to the above scenario and can be applied to various scenarios where a virtual scene needs to be constructed.

[0071] See Figure 3 , in the actual application process, to construct a virtual surgical scene, it is first necessary to obtain multi-dimensional information, including at least patient information, organ information, and surgical information. Among them, patient information includes the patient's basic information, physiological indicators, preoperative tracking, and taboo risks. It should be noted that the taboo risk information is determined by information extraction from the above three types of information and screening and matching in the knowledge graph; organ information includes angiography screening information and similar case information. At the same time, after information extraction of the above two types of information, the surgical scene information of cases similar to the patient can be determined based on the corresponding knowledge graph; further, by combining the surgical information with the preoperative preparation information containing the necessary steps and materials for the surgery, the surgical information can be obtained.

[0072] Continue to refer to Figure 3 After determining the patient information, organ information, and surgical information in the scene construction information, the above information can be processed in a templated manner based on semantic segmentation technology, so as to eliminate invalid and redundant information in the information. Further, the scene construction information is packaged into RDF triples or multi-tuples, and the struc2vec model is used for graph embedding processing, that is, converting a high-dimensional matrix into a low-dimensional feature sequence, so as to facilitate inputting the data into the algorithm model.

[0073] Continue to refer to Figure 3 After determining the to-be-processed feature sequence, it can be input into a pre-trained GAT model to obtain the target feature sequence. The surgical training system can construct a virtual surgical scene corresponding to the target user based on the output data. In the constructed virtual surgical scene, patient information, organ information, and surgical information can be simulated. At the same time, since the taboo risk information is integrated in the multi-dimensional scene construction information, unexpected situations during the surgical process can also be simulated in the scene. Finally, based on the operation information received by the surgical training system and the knowledge graph model retrieved in real time, real-time feedback on the effects generated during the surgical process can be provided. After the surgical training is completed, the surgical results can also be evaluated.

[0074] The beneficial effects of the above technical solution are as follows: The automated and efficient construction of the virtual surgical scene is realized. At the same time, by integrating multi-dimensional information during the scene construction process, the limitations of traditional mechanical devices and initial imaging parameters are eliminated, making the generated virtual surgical scene more diverse and more in line with the actual needs in the clinical scenario.

[0075] Embodiment 4

[0076] Figure 4 The following is a structural block diagram of a virtual scene construction device for surgery provided in Embodiment 4 of the present invention, which can execute the virtual scene construction method for surgery provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. As Figure 4 shown, the device specifically includes: a scene construction information determination module 310 and a virtual surgical scene determination module 320.

[0077] The scene construction information determination module 310 is used to determine the scene construction information associated with the target user and determine the target to-be-processed matrix of the scene construction information; wherein, the scene construction basic information includes the target basic information corresponding to the target user, at least one to-be-processed organ information associated with the target lesion, and the target historical diagnosis and treatment information associated with the target lesion.

[0078] The virtual surgical scene determination module 320 is configured to process the target matrix to be processed based on a pre-trained virtual scene construction model, determine a virtual surgical scene corresponding to the target user, and perform surgical simulation based on the virtual surgical scene.

[0079] Based on the above technical solutions, the scene construction information determination module 310 includes a target basic information determination unit, a to-be-processed organ information determination unit, and a target historical diagnosis and treatment information determination unit.

[0080] The target basic information determination unit is configured to obtain the to-be-processed basic parameters corresponding to the target user, and extract the target basic information corresponding to the preset fields from the to-be-processed basic parameters.

[0081] The to-be-processed organ information determination unit is configured to determine the at least one to-be-processed organ information from the to-be-processed views associated with the target lesion based on an image recognition algorithm.

[0082] The target historical diagnosis and treatment information determination unit is configured to determine the target historical diagnosis and treatment information associated with the target lesion in the historical diagnosis and treatment information database according to a pre-established diagnosis and treatment knowledge graph.

[0083] Based on the above technical solutions, the target basic information further includes target contraindication information corresponding to the target user.

[0084] Optionally, the target basic information determination unit is further configured to determine the target contraindication information corresponding to the target basic information according to a pre-established contraindication knowledge graph, and update the target contraindication information to the target basic information, so as to simulate intraoperative emergencies according to the target contraindication information.

[0085] Based on the above technical solutions, the scene construction information determination module 310 further includes a target matrix to be processed determination unit.

[0086] The target matrix to be processed determination unit is configured to splice the target basic information, the at least one to-be-processed organ information, and the target historical diagnosis and treatment information to construct the target matrix to be processed.

[0087] Based on the above technical solutions, the virtual scene construction model includes a node similarity sub-model and a graph attention network sub-model.

[0088] Based on the above technical solutions, the virtual surgical scene determination module 320 includes a to-be-processed feature sequence determination unit and a virtual scene construction unit.

[0089] A to-be-processed feature sequence determination unit, configured to perform dimensionality reduction processing on the target to-be-processed matrix based on the node similarity sub-model to obtain a corresponding to-be-processed feature sequence.

[0090] A virtual scene construction unit, configured to process the to-be-processed feature sequence based on a pre-trained graph attention network sub-model to obtain a target feature sequence, and construct the virtual surgical scene according to the target feature sequence; wherein, the virtual surgical scene includes virtual user information, virtual organ information, virtual diagnosis and treatment information, and intraoperative emergencies.

[0091] Optionally, the virtual scene construction unit is further configured to input the to-be-processed feature sequence into a pre-trained graph attention network sub-model to obtain attention coefficients of each element in the to-be-processed feature sequence respectively; and determine the target feature sequence according to each attention coefficient and the corresponding element.

[0092] Based on the above technical solutions, the virtual scene construction device applied to surgery further includes a model training module.

[0093] The model training module is configured to obtain a training sample set; wherein, the training sample set includes a plurality of training samples, and each training sample includes: historical scene construction information associated with a historical user and actual surgical scene information; use the historical scene construction information in each training sample as the input of the to-be-trained virtual scene construction model, use the actual surgical scene information as the output of the to-be-trained virtual scene construction model, and train the to-be-trained virtual scene construction model; use the loss convergence in the to-be-trained virtual scene construction model as the training target, and train to obtain the virtual scene construction model.

[0094] The technical solution provided in this embodiment first determines the scene construction information associated with the target user, and determines the target to-be-processed matrix of the scene construction information, so as to determine the target basic information of the target user, at least one to-be-processed organ information of the target lesion, and the historical diagnosis and treatment information of the target lesion; further, based on the pre-trained virtual scene construction model, process the target to-be-processed matrix to determine the virtual surgical scene corresponding to the target user, so as to perform surgical simulation based on the virtual surgical scene, realizing the automated and efficient construction of the virtual surgical scene. At the same time, by integrating information of multiple dimensions during the scene construction process, it gets rid of the limitations of traditional mechanical devices and initial imaging parameters, making the generated virtual surgical scene more diverse and more in line with the actual needs in the clinical scenario.

[0095] The virtual scene construction device applied to surgery provided in the embodiments of the present invention can execute the virtual scene construction method applied to surgery provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0096] It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present invention.

[0097] Embodiment Five

[0098] Figure 5 FIG. is a schematic structural diagram of an electronic device provided in Embodiment Five of the present invention. Figure 5 FIG. shows a block diagram of an exemplary electronic device 40 suitable for use in implementing the embodiments of the present invention. Figure 5 The electronic device 40 shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0099] As Figure 5 shown, the electronic device 40 is presented in the form of a general-purpose computing device. The components of the electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 connecting different system components (including the system memory 402 and the processing unit 401).

[0100] The bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0101] The electronic device 40 typically includes a variety of computer system-readable media. These media can be any available media accessible by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0102] The system memory 402 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. The electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 406 may be used for reading and writing non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive"). Although Figure 5Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) can be provided. In these cases, each drive can be connected to the bus 403 through one or more data medium interfaces. The memory 402 may include at least one program product having a set (for example, at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0103] A program / utility 408 having a set (at least one) of program modules 407 can be stored in, for example, the memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 407 generally perform the functions and / or methods in the embodiments described in the present invention.

[0104] The electronic device 40 can also communicate with one or more external devices 409 (such as a keyboard, a pointing device, a display 410, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or communicate with any device that enables the electronic device 40 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 411. Moreover, the electronic device 40 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 412. As shown in the figure, the network adapter 412 communicates with other modules of the electronic device 40 through the bus 403. It should be understood that although Figure 5 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0105] The processing unit 401 executes various functional applications and data processing by running the programs stored in the system memory 402, such as implementing the virtual scene construction method applied to surgery provided by the embodiments of the present invention.

[0106] Embodiment Six

[0107] Embodiment Six of the present invention also provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute a virtual scene construction method applied to surgery when executed by a computer processor.

[0108] The method includes:

[0109] Determine the scenario construction information associated with the target user, and determine the target matrix to be processed of the scenario construction information; wherein, the scenario construction basic information includes the target basic information corresponding to the target user, at least one organ information to be processed associated with the target lesion, and the target historical diagnosis and treatment information associated with the target lesion.

[0110] Process the target matrix to be processed based on a pre-trained virtual scenario construction model to determine a virtual surgical scenario corresponding to the target user, so as to perform surgical simulation based on the virtual surgical scenario.

[0111] The computer storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable item code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0113] The item code contained on the computer-readable medium may be transmitted by any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0114] Computer program code for performing the operations of the embodiments of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0115] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for constructing a virtual scenario applied to surgery, characterized in that Including: Determine scenario construction information associated with a target user, where the scenario construction information includes target basic information corresponding to the target user, at least one organ information to be processed associated with a target lesion, and target historical diagnosis and treatment information associated with the target lesion; Concatenate the target basic information, the at least one organ information to be processed, and the target historical diagnosis and treatment information to construct a target matrix to be processed; The target matrix to be processed is obtained by screening and sorting redundant information in the scenario construction information based on a preset template and then performing a packaging process; Process the target matrix to be processed based on a pre-trained virtual scenario construction model to determine a virtual surgical scenario corresponding to the target user, so as to perform surgical simulation based on the virtual surgical scenario; Wherein, the virtual scenario construction model includes a node similarity sub-model and a graph attention network sub-model, and processing the target matrix to be processed based on a pre-trained virtual scenario construction model to determine a virtual surgical scenario corresponding to the target user includes: Based on the node similarity sub-model, perform dimensionality reduction processing on the target matrix to be processed to obtain a corresponding feature sequence to be processed; the dimensionality reduction processing is to map the target matrix to be processed into a low-dimensional vector based on a graph embedding process; Process the feature sequence to be processed based on a pre-trained graph attention network sub-model to obtain a target feature sequence, and construct the virtual surgical scenario according to the target feature sequence; the virtual surgical scenario includes virtual user information, virtual organ information, virtual diagnosis and treatment information, and intraoperative emergencies.

2. The method according to claim 1, wherein The determining the scenario construction information associated with the target user includes: Obtain to-be-processed basic parameters corresponding to the target user, and extract target basic information corresponding to a preset field from the to-be-processed basic parameters; Based on an image recognition algorithm, determine the at least one organ information to be processed from to-be-processed views associated with the target lesion; According to a pre-established diagnosis and treatment knowledge graph, determine target historical diagnosis and treatment information associated with the target lesion in a historical diagnosis and treatment information database.

3. The method according to claim 2, wherein The target basic information further includes target taboo information corresponding to the target user, and the extracting the target basic information corresponding to a preset field from the to-be-processed basic parameters includes: According to a pre-established taboo knowledge graph, determine target taboo information corresponding to the target basic information, and update the target taboo information to the target basic information, so as to simulate intraoperative emergencies according to the target taboo information.

4. The method according to claim 1, characterized in that The processing the feature sequence to be processed based on a pre-trained graph attention network sub-model to obtain a target feature sequence includes: Input the feature sequence to be processed into a pre-trained graph attention network sub-model to obtain attention coefficients of each element in the feature sequence to be processed respectively; Determine a target feature sequence according to each attention coefficient and the corresponding element.

5. The method according to claim 1, characterized in that Before processing the target matrix to be processed based on a pre-trained virtual scenario construction model, it further includes: Obtain a training sample set; wherein, the training sample set includes a plurality of training samples, and each training sample includes: historical scenario construction information associated with a historical user and actual surgical scenario information; Use the historical scenario construction information in each training sample as the input of the virtual scenario construction model to be trained, and use the actual surgical scenario information as the output of the virtual scenario construction model to be trained, and train the virtual scenario construction model to be trained; Take the loss convergence in the virtual scenario construction model to be trained as the training objective, and train to obtain the virtual scenario construction model.

6. A virtual scene construction device applied to surgery, characterized in that, It includes: A scenario construction information determination module, configured to determine scenario construction information associated with a target user, where the scenario construction information includes target basic information corresponding to the target user, at least one organ to be processed associated with a target lesion, and target historical diagnosis and treatment information associated with the target lesion; The scenario construction information determination module further includes: A target matrix to be processed determination unit, configured to splice the target basic information, the at least one organ to be processed information, and the target historical diagnosis and treatment information to construct a target matrix to be processed; wherein, the target matrix to be processed is obtained by screening and sorting redundant information in the scenario construction information based on a preset template and then performing a packaging process; A virtual surgical scenario determination module, configured to process the target matrix to be processed based on a pre-trained virtual scenario construction model, and determine a virtual surgical scenario corresponding to the target user, so as to perform surgical simulation based on the virtual surgical scenario; The virtual scenario construction model includes a node similarity sub-model and a graph attention network sub-model, and the virtual surgical scenario determination module includes: A feature sequence to be processed determination unit, configured to perform dimensionality reduction processing on the target matrix to be processed based on the node similarity sub-model to obtain a corresponding feature sequence to be processed; the dimensionality reduction processing is to map the target matrix to be processed into a low-dimensional vector based on a graph embedding process; A virtual scenario construction unit, configured to process the feature sequence to be processed based on a pre-trained graph attention network sub-model to obtain a target feature sequence, and construct the virtual surgical scenario according to the target feature sequence; the virtual surgical scenario includes virtual user information, virtual organ information, virtual diagnosis and treatment information, and intraoperative emergencies.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual scenario construction method for surgery according to any one of claims 1-5.

8. A storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute the virtual scenario construction method for surgery according to any one of claims 1-5 when executed by a computer processor.

Images