Object relation identification method and device applied to operating room scene, electronic equipment, storage medium and program product

Abstract

The invention provides an object relation identification method and device applied to an operating room scene, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a plurality of operation images of an operating room, the plurality of operation images are obtained through the shooting of a plurality of image sensors at the same time, and the image sensors are used for carrying out the shooting of the plurality of operation images; the plurality of image sensors are arranged in different directions in the operating room and are used for capturing an operation process; according to a preset grid template, splicing the plurality of operation images to obtain an operating room image; and performing object relation identification processing on the operating room image to obtain object relation information among different objects in the operating room. According to the method, the operation images of multiple visual angles are spliced into one operation room image according to the preset grid template, rich details in the operation room can be conveniently and intensively collected, a large amount of time is not needed to construct three-dimensional point clouds, the object relation recognition processing time can be effectively saved, and the real-time requirements of scene understanding and analysis in the operation are met.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing technology, and more specifically, to an object relationship recognition method, device, electronic device, storage medium and program product applied to an operating room scene. Background Art

[0002] Understanding the operating room scene is of great significance to surgery. Artificial intelligence algorithms achieve understanding of the operating room scene by analyzing the spatial occupancy relationship between doctors, the spatial relationship between doctors and equipment, and the spatial relationship between equipment in the operating room, thereby providing early warnings for incorrect positioning and environment during surgery, optimizing the surgical process, improving surgical efficiency, and improving the quality of surgical treatment.

[0003] In the related art, there is a solution for understanding the scene in the operating room based on a multimodal large model. This solution first extracts visual features from the 3D point cloud and RGB image simultaneously, and then aligns and fine-tunes the visual features with the large language model to complete the understanding of the real-world operating room scene relationship. However, in real-world applications, it takes a long time to reconstruct the scene point cloud from multi-view video streams, which is difficult to meet the real-time requirements of scene understanding and analysis during surgery. Summary of the invention

[0004] The present disclosure provides an object relationship recognition method, device, electronic device, storage medium and program product applied to an operating room scene, for solving at least one of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an object relationship recognition method applied to an operating room scene, the object relationship recognition method comprising: acquiring a plurality of surgical images of an operating room, wherein the plurality of surgical images are captured at the same time by a plurality of image sensors, and the plurality of image sensors are arranged at different positions in the operating room and are used to capture the surgical process; performing splicing processing on the plurality of surgical images according to a preset grid template to obtain an operating room image; and performing object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room.

[0006] Optionally, the performing object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room includes: extracting image features of the operating room image; obtaining question text regarding the relationship between different objects in the operating room; and using a large language model to process the image features and the question text to obtain object relationship information between different objects in the operating room.

[0007] Optionally, extracting image features of the operating room image includes: extracting image features of the operating room image using a CLIP model.

[0008] Optionally, the large language model includes an LLaVA model based on the LLaMA series of models.

[0009] Optionally, the use of a large language model to process the image features and the question text to obtain object relationship information between different objects in the operating room includes: converting the image features into one-dimensional image features, and mapping the one-dimensional image features into a language space to obtain language features; converting the question text into a one-dimensional word segmentation sequence; merging the language features and the word segmentation sequence into a sequence, and passing the sequence into the large language model to obtain the object relationship information between different objects in the operating room output by the large language model.

[0010] Optionally, the large language model is trained using a supervised method for operating room scenarios; and the CLIP model is trained using a supervised method for operating room scenarios.

[0011] According to a second aspect of an embodiment of the present disclosure, there is provided an object relationship recognition device for use in an operating room scene, the object relationship recognition device comprising: an acquisition unit, configured to acquire a plurality of surgical images of the operating room, wherein the plurality of surgical images are captured at the same time by a plurality of image sensors, and the plurality of image sensors are arranged at different positions in the operating room and are used to capture the surgical process; a merging unit, configured to perform splicing processing on the plurality of surgical images according to a preset grid template to obtain an operating room image; and a processing unit, configured to perform object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room.

[0012] Optionally, the processing unit is further configured to: extract image features of the operating room image; obtain question text regarding the relationship between different objects in the operating room; and use a large language model to process the image features and the question text to obtain object relationship information between different objects in the operating room.

[0013] Optionally, the processing unit is further configured to extract image features of the operating room image using a CLIP model.

[0014] Optionally, the large language model includes an LLaVA model based on the LLaMA series of models.

[0015] Optionally, the processing unit is further configured to: convert the image features into one-dimensional image features, and map the one-dimensional image features into a language space to obtain language features; convert the question text into a one-dimensional word segmentation sequence; merge the language features and the word segmentation sequence into a sequence, and pass it into the large language model to obtain object relationship information between different objects in the operating room output by the large language model.

[0016] Optionally, the large language model is trained using a supervised method for operating room scenarios; and the CLIP model is trained using a supervised method for operating room scenarios.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute an object relationship recognition method applied to an operating room scene according to an exemplary embodiment of the present disclosure.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute an object relationship recognition method applied to an operating room scene according to an exemplary embodiment of the present disclosure.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute an object relationship recognition method applied to an operating room scene according to an exemplary embodiment of the present disclosure.

[0020] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0021] According to the object relationship recognition method and device, electronic device, and storage medium applied to the operating room scene disclosed in the present invention, by splicing surgical images from multiple perspectives into one operating room image according to a preset grid template, rich details in the operating room can be collected conveniently and centrally without spending a lot of time to construct a three-dimensional point cloud, thereby effectively saving the time for object relationship recognition processing and meeting the real-time requirements of scene understanding and analysis during surgery.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0024] Figure 1 is a flow chart of an object relationship recognition method applied to an operating room scene according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of an operating room image according to a specific embodiment of the present disclosure;

[0026] Figure 3 is a block diagram of an object relationship recognition device applied to an operating room scene according to an exemplary embodiment of the present disclosure;

[0027] Figure 4 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0030] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0031] Hereinafter, an object relationship recognition method and apparatus, an electronic device, and a storage medium applied to an operating room scene according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0032] Figure 1 This is a flow chart of an object relationship recognition method applied to an operating room scene according to an exemplary embodiment of the present disclosure. It should be understood that the objects in the operating room scene may include characters such as doctors, nurses, and patients in the operating room, and may also include objects such as light sources, tables and chairs, and surgical tools. The object relationship recognition method applied to the operating room scene can be executed by an electronic device with sufficient computing power.

[0033] Reference Figure 1 In step S101, a plurality of surgical images of an operating room are acquired.

[0034] Multiple surgical images are captured by multiple image sensors at the same time, and the multiple image sensors are arranged in different positions in the operating room and are used to capture the surgical process. These multiple surgical images can capture images in the surgical process from different perspectives in all directions. Specifically, these multiple image sensors can be connected to an electronic device that executes the object relationship recognition method of the exemplary embodiment of the present disclosure through a communication network, so as to transmit the captured surgical images to the electronic device. As an example, the shooting of multiple image sensors can be uniformly controlled by an electronic device to improve synchronization; multiple image sensors can also shoot surgical images at a preset period and transmit them to the electronic device; multiple image sensors can also shoot videos (for example, they can shoot videos at the same frame rate), and then send the video streams to the electronic device, and the electronic device aligns the time of the multiple video streams, and extracts the image frames at the same time from the multiple video streams, as multiple surgical images at the same time, and the present disclosure does not limit this. It should be understood that the "same time" here is not the same time in a strict sense, and reasonable errors are allowed due to technical limitations such as control accuracy and communication delay.

[0035] In step S102, multiple surgical images are spliced ​​according to a preset grid template to obtain an operating room image.

[0036] The preset grid template is used to specify the positions at which multiple different images are stitched together, providing a unified stitching standard for operating room images, ensuring the consistency and stability of the stitching format of operating room images, helping to reduce recognition errors caused by different stitching formats during subsequent object relationship recognition processing, and helping to improve recognition efficiency.

[0037] Regarding the preset grid template, as an example, only the number of rows or columns of the grid may be required, and then the number of columns or rows of the grid may be flexibly adjusted according to the actual number of surgical images. At this time, if the number of surgical images is not an integer multiple of the number of rows or columns of the grid, resulting in redundant cells (i.e., cells in the grid), a predetermined meaningless image (e.g., a pure white image) may be used to fill the redundant cells, thereby forming a complete operating room image; the number of surgical images may also be set in advance, and the preset grid template may be designed according to the number. Furthermore, if the viewing angle of each surgical image is set in advance, the preset grid template may further specifically define the viewing angle corresponding to each cell, thereby placing each surgical image at the cell of the corresponding viewing angle during splicing. The present disclosure does not limit the specific form of the preset grid template.

[0038] In the process of performing the stitching process, as an example, the acquired multiple surgical images can be directly stitched according to the preset grid template, or the multiple surgical images can be preprocessed first, and then the multiple images obtained by the preprocessing are stitched according to the preset grid template. The preprocessing includes, for example, but is not limited to, image alignment and perspective conversion to improve the image quality and ensure that the final generated operating room image can clearly reflect the actual situation of the surgical scene. The present disclosure does not limit the specific stitching process.

[0039] In step S103, object relationship recognition processing is performed on the operating room image to obtain object relationship information between different objects in the operating room.

[0040] According to the object relationship recognition method applied to the operating room scene of the exemplary embodiment of the present disclosure, by splicing surgical images from multiple perspectives into one operating room image according to a preset grid template, it is possible to conveniently and centrally collect rich details in the operating room without spending a lot of time to construct a three-dimensional point cloud, thereby effectively saving the time for object relationship recognition processing and meeting the real-time requirements of scene understanding and analysis during surgery.

[0041] Next, how to perform the object relationship identification process in step S103 will be further introduced.

[0042] In some embodiments, optionally, step S103 can process the operating room image based on a series of small visual models such as ResNet and PointNet. In this case, it is necessary to first use the object detection model to perform object detection of doctors and equipment, and then use the relationship recognition model to perform relationship recognition of the detected objects to obtain object relationship information between different objects in the operating room.

[0043] In other embodiments, optionally, step S103 includes: extracting image features of the operating room image; obtaining question text regarding the relationship between different objects in the operating room; using a large language model to process the image features and question text to obtain object relationship information between different objects in the operating room. By using a large language model to process the image features and question text of the operating room image, the powerful semantic understanding and language processing capabilities of the large language model can be utilized to more reliably understand the meaning of the operating room image, and convert the object relationship recognition problem in the image into a semantic understanding and answering problem, which helps to greatly improve the efficiency of information processing, improve the response efficiency of object relationship recognition processing, and reliably meet the real-time requirements of scene understanding and parsing during surgery. In addition, because the large language model itself has strong adaptability, the object relationship recognition scheme of this embodiment can well adapt to complex tasks in the real world and open environment.

[0044] As an example, the large language model may adopt a multimodal large language model to simultaneously process image features and question text.

[0045] In some other embodiments described above, regarding the operation of extracting image features of operating room images, further optionally, the operation includes: using an image semantic analysis method based on deep learning, such as but not limited to using a CLIP (Contrastive Language-Image Pretraining) model to extract image features of operating room images. The image semantic analysis method based on deep learning converts an image into words or a text to describe it, uses a deep convolutional neural network and a deep autoencoder to extract multi-layer features of the image, and extracts the language description (visual word) of the image based on this, establishes an inverted index, and generates an effective and accurate image search method. Specifically, the CLIP model is a multimodal model that can effectively pair images and texts. By specifically using the CLIP model to verbalize operating room images, efficient extraction of image features of operating room images can be achieved, so that the extracted image features contain language information.

[0046] Optionally, the CLIP model is trained using a supervised method for operating room scenes, so that the CLIP model can understand the operating room scenes more specifically, which helps to improve the accurate extraction of image features of operating room images. Specifically, the training is based on the pre-trained CLIP model and is fine-tuned for operating room scenes. When the CLIP model is trained using a supervised method, its training samples may include sample operating room images and pre-labeled real image features. The sample operating room images may be obtained by splicing sample surgical images of multiple different perspectives according to a preset grid model.

[0047] In some other embodiments described above, regarding the operation of using a large language model to process image features and question text to obtain object relationship information between different objects in the operating room, further optionally, the large language model used includes an LLaVA (Large Language and Vision Assistant) model based on the LLaMA (Large Language Model Meta AI) series of models. The LLaMA series of models aims to improve natural language processing capabilities by pre-training on a wide range of text data. The LLaVA model is a model that combines language and visual processing capabilities, can understand and generate text related to images, supports multimodal interaction, and can provide a comprehensive understanding of the scene. The LLaVA model based on the LLaMA series of models refers to an LLaVA model that uses the LLaMA series of models when processing text. By using this model, the image features of the operating room image and the question text regarding the relationship between different objects in the operating room can be fully processed, so that the questions are answered based on the image features to obtain object relationship information. As an example, a LLaVA model based on LLaMA3 can be used specifically. LLaMA3 is the latest version of the LLaMA series, with powerful language understanding and generation capabilities. This model was developed by the MetaAI team to provide a lightweight but powerful language model that enables a wider research community and developers to access and utilize the most advanced technologies. The open source nature of the LLaMA series of models allows individuals and companies to explore and innovate freely without being restricted by high costs, which can effectively reduce the application cost of the solution.

[0048] Specifically, the operation may include: converting image features into one-dimensional image features, and mapping the one-dimensional image features into the language space to obtain language features; converting the question text into a one-dimensional word segmentation sequence; merging the language features and the word segmentation sequence into a sequence, and passing it into the large language model to obtain the object relationship information between different objects in the operating room output by the large language model. By converting the image features and the question text into one-dimensional vectors respectively, and mapping the one-dimensional image features into the language space, a language feature that can be understood and processed by the large language model is obtained, and then the vectors are merged, for example, two one-dimensional vectors are spliced ​​into a one-dimensional vector with a higher dimension, and the input data of the large language model can be obtained, which ensures the smooth processing of the large language model. As an example, for language space mapping, a multi-layer perceptron (MLP) can be used to achieve it, and a one-dimensional language feature is obtained, thereby converting the visual information into a data format that can be directly used by the large language model.

[0049] Optionally, the large language model is trained using a supervised method for an operating room scene, so that the large language model can understand the operating room scene more specifically, which helps to improve the accurate extraction of object relationships between different objects in the operating room. Specifically, the training is based on a pre-trained large language model and is fine-tuned for an operating room scene. When using a supervised method to train a large language model, its training samples may include image features of sample operating room images and pre-labeled real object relationship information. The sample operating room images may be obtained by splicing sample surgical images of multiple different perspectives according to a preset grid model. A network for converting image features into language features (such as the above-mentioned multilayer perceptron) can be trained together with the large language model.

[0050] Next, an object relationship recognition method applied to an operating room scene according to a specific embodiment of the present disclosure is introduced.

[0051] The purpose of this specific embodiment is to solve two key problems in the recognition of person-object relationships in operating room scenes: (1) how to understand multi-view operating room scene images; and (2) how to represent and generate relationships between people and objects.

[0052] To this end, this specific embodiment provides a multi-view scene relationship understanding network structure based on a multimodal large language model, including a storage module, an image processing module and a large language model module. The storage module is responsible for storing the image data of various objects in the operating room and related metadata. The metadata is specifically the original video or image captured by the multi-view camera, and transmits the multi-view surgical images to the image processing module. The image processing module receives the multi-view surgical images from the storage module, performs necessary preprocessing operations, obtains the operating room images, and extracts the high-dimensional features of the operating room images based on the pre-trained visual module. The high-dimensional features of the image will be converted into one-dimensional language features, integrated with the question text, and processed by the large language model module to finally generate the character-object relationship in the scene.

[0053] In specific practice, the following prompt words can be input into a comprehensive large language model to guide the model to execute the object recognition method of this specific embodiment.

[0054] Prompt:

[0055] You want to complete a brief description of the instructions for using a large language model to identify object relationships in an operating room scene. It includes: storage module, image processing module and large language model module. The large language model uses the LLaVA model based on the LLaMA3 model. The method is mainly divided into six steps:

[0056] 1. Collect multi-view images of the operating room.

[0057] 2. Integrate multi-view images into a single image.

[0058] 3. Input the integrated image into the pre-trained CLIP model to extract image features.

[0059] 4. The image features are input into the visual-language modality alignment module of the multimodal large model. Through modality alignment, the image features are translated into the language space to obtain the language features of the image.

[0060] 5. Input the linguistic features of the image and questions about the relationships in the scene into the fine-tuned large language model.

[0061] 6. The large language model generates the relationships between people and objects in the scene.

[0062] After the prompt word is input, the integrated large language model can execute the object recognition method of this specific embodiment. Corresponding to the prompt word, the method includes the following 6 steps:

[0063] Step S1, collect multi-view images of the operating room and store them in a storage module. Multiple high-resolution cameras are deployed in the operating room to capture images of the surgical process from different angles (corresponding to the surgical images mentioned above). These multi-view images can provide a comprehensive view of the surgical scene, including the dynamics of the surgical team, the use of surgical tools, and the patient's condition.

[0064] Step S2: Integrate the multi-view images into a single image. For a large language model, processing multi-view images simultaneously is an extremely complex task, so the information of multiple images needs to be integrated. Here, for example Figure 2 As shown in the figure, six perspective images (images taken from six different directions in the operating room toward the direction of the operating table) are spliced ​​into a three-row and two-column grid image by the image processing module, corresponding to the operating room image mentioned above. This seemingly simple method has many advantages. First, the images from multiple perspectives are integrated into one image, which is convenient for subsequent processing. Second, the information of multiple images will not be lost during the integration process.

[0065] Step S3, input the integrated image into the pre-trained CLIP model to extract the image features of the image. The image processing module also uses the pre-trained CLIP model to analyze the integrated image and extract image features related to the image content. The CLIP model learns to understand the relationship between these two different modalities by jointly training images and texts on a large-scale data set. This enables the CLIP model to better understand and interpret the image content and its relationship with the text description, so that the extracted image features can contain more accurate language information. At the same time, the CLIP model demonstrates excellent generalization capabilities. It can effectively handle new scenes or objects that have not directly appeared in the training data, which is especially important in specific scenes such as operating rooms because the surgical environment may have many variables.

[0066] Step S4, maps the image features to the features of the large language model as language features. The image processing module also converts the image features extracted by the CLIP model into new feature vectors, which can be understood and processed by the large language model. This step is to convert the visual information into a data format that can be directly used by the large language model. At the same time, as features used to describe the image, the image features are still expressed in two dimensions with a size of H×W, where H and W represent the height and width of the image, respectively. The two-dimensional image features will be flattened into one-dimensional features with a length equal to H·W, and each one-dimensional element represents the feature of a pixel in the two-dimensional image. The mapper here is a multi-layer perceptron (MLP), and its mapping can be expressed as:

[0067] f llm =MLP(f clip ).

[0068] Among them, f llm Represents the language features after mapping, f clip Represents the features before mapping, that is, one-dimensional image features, and both have a length of H·W.

[0069] Step S5, input the language features and the questions about the relationships in the scene into the fine-tuned large language model. The converted feature vector and the relationship questions are input into the large language model that has been fine-tuned for the medical surgery scene. This step uses the model's powerful language understanding ability to analyze and reason about the object relationships in the surgical scene.

[0070] Specifically, the large language model requires input and output expressed in sequences. Relational questions can be raised by users, such as "What are the relationships among all doctors in the operating room?", "What are all the scene relationships about the surgeon in the operating room?", "What are all the relationships that exist in the operating room?". A unified relational question "What are the relationships among humans and instruments in the given images?" is used here. In order for the large language model to understand the language, this question will be passed to the tokenizer and represented using a token sequence. The one-dimensional language features and the one-dimensional token sequence are merged into a sequence and passed to the large language model.

[0071] Step S6: The large language model generates the relationship between people and objects in the scene. The large language model generates a report describing the relationship between objects and people in the surgical scene in detail based on the input features and questions.

[0072] The output form of the large language model is a probability sequence, which corresponds to a word segmentation sequence representing an object relationship. Each element in the object relationship word segmentation sequence represents the relationship between two objects, which can be used <obj a1 > <relationship a > <obj a2 > in the form of object obj a1 and obj a2 There is a relationship between a This relationship. Each element in the probability sequence is a one-hot encoding, which is used to represent the probability of the corresponding element in the word segmentation sequence.

[0073] In addition, the large language model involved in step S4 to step S6 is trained using a supervised method for the operating room scene. The training process is as follows:

[0074] First, in the data preparation stage, for sample operating room images that are pre-labeled with real object relationship information, the real object relationship information must first be rewritten into the output form of the large language model so that it can be used as a benchmark and compared with the object relationship information output by the large language model, and the pre-trained large language model can be fine-tuned accordingly.

[0075] The relationship between a group of people or objects can be expressed by a ternary relationship group, for example:

[0076] [ <obj1> , <relationship> , <obj2>].

[0077] in, <obj1>and <obj2>represents people or objects in the scene, and <relationship>It indicates the relationship between people or objects.

[0078] The ternary relationship groups in the scene will be rewritten into text segments, for example:

[0079] <obj a1 > <relationship a > <obj a2 >\n

[0080] <obj b1 > <relationship b > <obj b2 >\n

[0081] …

[0082] Each element in a triple relationship group is separated by a space, and each triple relationship group is separated by a line break \n. The text segment of the triple relationship group is also passed to the tokenizer and represented as an object relationship token sequence. Each token is represented in the form of a one-hot encoding based on the real object relationship information, and also represents the fact probability of each token being predicted, and the fact probability sequence is obtained:

[0083]

[0084] Then, each time the parameters are adjusted in the fine-tuning stage, the large language model to be trained is first used to perform the above steps S3 to S6 on the sample operating room images to obtain the probability sequence output by the model. The large language model to be trained specifically uses the pre-trained LLaVA model based on LLaMA3, and the main architecture is Transformer. The probability sequence output by the model is recorded as the predicted probability sequence:

[0085] [t1,t2,t3,…,t n ].

[0086] Each element in the predicted probability sequence is the probability of the triple relationship group corresponding to each word segmentation predicted by the model.

[0087] Then, for the fact probability sequence and the predicted probability sequence, the cross entropy is calculated as the loss to update the network parameters of the large language model. ce It can be expressed as:

[0088]

[0089] Among them, the large language model is updated using LoRA (Low-Rank Adaptation), and the mapper mentioned in sub-step S4 also participates in the update, and is specifically updated using back propagation.

[0090] This specific embodiment can provide an in-depth understanding of complex surgical scenarios by efficiently combining multi-view image processing technology and advanced large language model technology, thereby helping the medical team make more accurate decisions during surgery and improving the success rate of surgery.

[0091] Figure 3 is a block diagram of an object relationship recognition device applied to an operating room scene according to an exemplary embodiment of the present disclosure. Figure 3 The object relationship identification device 300 includes an acquisition unit 301, a merging unit 302, and a processing unit 303.

[0092] The acquisition unit 301 can acquire multiple surgical images of the operating room, wherein the multiple surgical images are captured by multiple image sensors at the same time, and the multiple image sensors are arranged at different positions in the operating room and are used to capture the surgical process.

[0093] The merging unit 302 may perform splicing processing on multiple surgical images according to a preset grid template to obtain an operating room image.

[0094] The processing unit 303 may perform object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room.

[0095] Optionally, the processing unit 303 can also: extract image features of the operating room image; obtain question text regarding the relationship between different objects in the operating room; use a large language model to process the image features and the question text to obtain object relationship information between different objects in the operating room.

[0096] Optionally, the processing unit 303 may also use a CLIP model to extract image features of the operating room image.

[0097] Optionally, the large language model includes an LLaVA model based on the LLaMA series of models.

[0098] Optionally, the processing unit 303 can also: convert image features into one-dimensional image features, and map the one-dimensional image features into the language space to obtain language features; convert the question text into a one-dimensional word segmentation sequence; merge the language features and the word segmentation sequence into one sequence, and pass it into the large language model to obtain the object relationship information between different objects in the operating room output by the large language model.

[0099] Optionally, the large language model is trained using a supervised method for operating room scenarios; the CLIP model is trained using a supervised method for operating room scenarios.

[0100] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0101] Figure 4 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0102] Reference Figure 4 The electronic device 400 includes: at least one memory 401 and at least one processor 402, wherein the at least one memory 401 stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor 402, the at least one processor is prompted to execute the target corresponding method as described in the above exemplary embodiment.

[0103] As an example, the electronic device 400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 400 is not necessarily a single electronic device 400, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 400 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device 400 interconnected with a local or remote (e.g., via wireless transmission) interface.

[0104] In the electronic device 400, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 402 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0105] The processor 402 may execute instructions or codes stored in the memory 401, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0106] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 402 can read files stored in the memory.

[0107] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0108] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, cause the at least one processor to execute the target corresponding method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0109] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including computer instructions, which, when executed by at least one processor, execute the target corresponding method as described in the above exemplary embodiment.

[0110] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0111] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.< / relationship> < / relationship> < / obj1>

Claims

1. An object relationship recognition method applied to an operating room scene, characterized in that: The object relationship identification method comprises: Acquire a plurality of surgical images in an operating room, wherein the plurality of surgical images are captured by a plurality of image sensors at the same time, and the plurality of image sensors are arranged at different positions in the operating room and are used to capture the surgical process; According to a preset grid template, the plurality of surgical images are spliced ​​to obtain an operating room image; Object relationship recognition processing is performed on the operating room image to obtain object relationship information between different objects in the operating room.

2. The object relationship identification method according to claim 1, characterized in that: The performing object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room includes: Extracting image features of the operating room image; Get the text of questions about the relationship between different objects in the operating room; The image features and the question text are processed using a large language model to obtain object relationship information between different objects in the operating room.

3. The object relationship identification method according to claim 2, characterized in that: The step of extracting image features of the operating room image comprises: The CLIP model is used to extract image features of the operating room image.

4. The object relationship identification method according to claim 2, characterized in that: The large language model includes an LLaVA model based on the LLaMA series of models.

5. The object relationship identification method according to claim 2, characterized in that: The use of a large language model to process the image features and the question text to obtain object relationship information between different objects in the operating room includes: Converting the image features into one-dimensional image features, and mapping the one-dimensional image features into a language space to obtain language features; Convert the question text into a one-dimensional word segmentation sequence; The language features and the word segmentation sequence are combined into one sequence, and the sequence is input into the large language model to obtain the object relationship information between different objects in the operating room output by the large language model.

6. The object relationship identification method according to claim 3, characterized in that: The large language model is trained using a supervised method for operating room scenarios; The CLIP model is trained using a supervised method for operating room scenarios.

7. An object relationship recognition device applied to an operating room scene, characterized in that: The object relationship identification device comprises: An acquisition unit is configured to acquire a plurality of surgical images of an operating room, wherein the plurality of surgical images are captured by a plurality of image sensors at the same time, and the plurality of image sensors are arranged at different positions in the operating room and are used to capture the surgical process; A merging unit is configured to perform splicing processing on the multiple surgical images according to a preset grid template to obtain an operating room image; The processing unit is configured to perform object relationship recognition processing on the operating room image to obtain object relationship information between different objects in the operating room.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the object relationship recognition method applied to the operating room scene as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the object relationship recognition method applied to an operating room scene according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute the object relationship recognition method applied to an operating room scene according to any one of claims 1 to 6.

Images