Method, device and storage medium for generating device usage data

Abstract

The present application relates to a method, device and storage medium for generating instrument usage data, wherein the method for generating instrument usage data comprises: obtaining a frame image to be tested, determining the category information of a target instrument identified from each frame image to be tested; calculating the first position information of the target instrument in the current frame image in the next frame image, and obtaining the second position information of the target instrument in the next frame image in the next frame image; wherein the frame image to be tested includes the current frame image and the next frame image; based on the first position information, the second position information and the category information, determining the usage status data of each target instrument; and generating the usage data of each target instrument according to the usage status data. Through the present application, the problem of low efficiency and accuracy in generating instrument usage data is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device and storage medium for generating instrument usage data. Background Art

[0002] Accurately recording the target instrument usage data during surgery is crucial to improving surgical safety, efficiency, and medical quality. Recording the target instrument usage data can provide data feedback to the surgical team, help optimize the surgical process, and provide strong support for medical quality control and surgical training through long-term data analysis. However, in related technologies, instrument usage data management is still in a manual or semi-automatic state. This process is not only time-consuming and labor-intensive, but also prone to errors, which greatly affects the efficiency and accuracy of target instrument usage data generation.

[0003] Currently, no effective solution has been proposed to address the problems of low efficiency and accuracy in generating device usage data in related technologies. Summary of the invention

[0004] The embodiments of the present application provide a method, device and storage medium for generating instrument usage data, so as to at least solve the problem of low efficiency and accuracy in generating instrument usage data in the related art.

[0005] In a first aspect, an embodiment of the present application provides a method for generating device usage data, the method comprising:

[0006] Acquire the frame images to be tested, and determine the category information of the target device identified from each of the frame images to be tested;

[0007] Calculating first position information of a target device in a current frame image in a next frame image, and obtaining second position information of a target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image;

[0008] determining usage status data of each of the target devices based on the first location information, the second location information, and the category information;

[0009] Based on the usage status data, usage data of each of the target devices is generated.

[0010] In some embodiments, the step of calculating the first position information of the target device in the current frame image in the next frame image includes:

[0011] Determine the current frame position information corresponding to the target device in the current frame image;

[0012] Performing a segmentation process on the current frame image according to the current frame position information to obtain a regional frame image;

[0013] The regional frame image and the next frame image are input into a trained target tracking model; using the target tracking model, a regional position matching the regional frame image is searched in the next frame image, and the first position information is determined based on the regional position.

[0014] In some embodiments, determining the usage status data of the target device based on the first location information, the second location information and the category information includes:

[0015] Based on the first position information, determining that the target device of the current frame image is in a first target area of ​​the next frame image, and based on the second position information, determining that the target device of the next frame image is in a second target area of ​​the next frame image;

[0016] Determine the degree of overlap between the first target area and the second target area; and determine the usage status data based on the degree of overlap and the category information.

[0017] In some embodiments, determining the degree of overlap between the first target area and the second target area includes:

[0018] The intersection result of the device area between the first target area and the second target area is calculated, and the overlap degree is determined according to the intersection result of the device area.

[0019] In some embodiments, determining the usage status data of each target device based on the first location information, the second location information and the category information includes:

[0020] Based on the first position information and the second position information, detecting whether the target device of the next frame image has been used in the current frame, and generating a use mark of the target device of the next frame image based on the detection result;

[0021] The frame images to be tested are traversed, and the usage status data of each target device is determined according to the usage mark and the category information corresponding to the target device of each frame image to be tested.

[0022] In some embodiments, determining the usage status data of each target device according to the usage mark and the category information corresponding to the target device of each frame image to be tested includes:

[0023] If the use mark indicates that the target device of the current frame image is not used, creating a corresponding device instance based on the target device of the next frame image;

[0024] The usage status data is determined based on the category information and the device instance.

[0025] In some embodiments, determining the usage status data according to the category information and the device instance includes:

[0026] Based on the category information, target devices with the same name among the target devices are determined, and device instances corresponding to the target devices with the same name are merged to obtain the usage status data.

[0027] In some embodiments, the step of acquiring the image frame to be tested includes:

[0028] Get the initial video stream;

[0029] Scene recognition is performed on the initial video stream to determine a target frame image in the initial video stream that is in a target scene, and the frame image to be tested is determined based on the target frame image.

[0030] In some embodiments, generating usage data of each target device according to the usage status data includes:

[0031] According to the usage status data, calculating the number of frames in which the corresponding target device appears in the frame image to be tested;

[0032] Calculating the usage time of the target device based on the number of occurrence frames, and calculating the total usage time ratio of the target device according to the usage time and the acquired total usage time;

[0033] Determine the number of consecutive appearances of the corresponding target device in the frame image to be tested according to the usage status data; determine the number of times the target device is used according to the consecutive number of appearances;

[0034] The usage data includes the usage time, the percentage of the total usage time and the number of times of usage.

[0035] In some embodiments, the usage data is expressed in the form of an array.

[0036] In a second aspect, an embodiment of the present application provides a device for generating device usage data, including:

[0037] An identification module, used to obtain the frame image to be tested and determine the category information of the target device identified from each of the frame images to be tested;

[0038] A position tracking module, used to calculate the first position information of the target device in the current frame image in the next frame image, and obtain the second position information of the target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image;

[0039] A usage status determination module, configured to determine usage status data of each of the target devices based on the first location information, the second location information and the category information;

[0040] A generating module is used to generate usage data of each target device according to the usage status data.

[0041] In a third aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating instrument usage data as described in the first aspect above.

[0042] Compared with the related art, the method, device and storage medium for generating instrument usage data provided in the embodiments of the present application obtain the frame image to be tested, determine the category information of the target instrument identified from each frame image to be tested; calculate the first position information of the target instrument in the current frame image in the next frame image, and obtain the second position information of the target instrument in the next frame image in the next frame image; wherein the frame image to be tested includes the current frame image and the next frame image; based on the first position information, the second position information and the category information, determine the usage status data of each target instrument; based on the usage status data, generate the usage data of each target instrument. Based on this, the present application can not only identify all instruments in the operating room, but also conduct a specific analysis of the usage of each target instrument, thereby providing an automated method for recording instrument usage data based on artificial intelligence and computer vision technology, eliminating the need for manual statistics and recording of instrument usage data, and effectively improving the efficiency and accuracy of instrument usage data generation.

[0043] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 It is a hardware structure block diagram of a terminal of a method for generating device usage data according to an embodiment of the present application;

[0046] Figure 2is a flow chart of a method for generating device usage data according to an embodiment of the present application;

[0047] Figure 3 is a flow chart of another method for generating device usage data according to an embodiment of the present application;

[0048] Figure 4 It is a structural block diagram of a device for generating equipment usage data according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for ordinary technicians in the field related to the contents disclosed in the present application, some changes such as design, manufacturing or production based on the technical contents disclosed in the present application are only conventional technical means, and should not be understood as insufficient contents disclosed in the present application.

[0050] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0051] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application means greater than or equal to two. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The terms "first", "second", "third" and the like involved in the present application are merely used to distinguish similar objects and do not represent a specific ordering of the objects.

[0052] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. Taking running on a terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a terminal of a method for generating device usage data according to an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0053] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating device usage data in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0055] This embodiment provides a method for generating device usage data. Figure 2 is a flow chart of a method for generating device usage data according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0056] Step S210 , obtaining the image frames to be tested, and determining the category information of the target device identified from each image frame to be tested.

[0057] The frame images to be tested are multiple frames of images generated by parsing a recorded video stream and collected by an image acquisition device such as a camera or a video recorder arranged at the surgical site. It should be noted that, in this embodiment, the image acquisition device can be set to collect a video from the beginning to the end of the surgery, and each frame in the surgical video is read out and saved as an image file in chronological order, and the frame rate of the video is marked as FPS, thereby obtaining the frame images to be tested.

[0058] Alternatively, in another embodiment, in order to ensure the integrity of surgical monitoring, an image acquisition device may also capture a complete video stream including preoperative scenes, intraoperative scenes, and postoperative scenes. In this case, scene recognition needs to be performed on the video stream, and the process of scene recognition to obtain the frame image to be tested may specifically include the following steps:

[0059] Obtain an initial video stream; perform scene recognition on the initial video stream, determine the target frame image in the initial video stream that is in the target scene, and determine the frame image to be tested based on the target frame image. The target scene refers to the usage scene of the target instrument. Taking the application scenario of identifying surgical instruments as an example, the initial video stream contains video stream data of the non-surgical part and video stream data of the surgical part; the non-surgical part usually includes preoperative scenes and postoperative scenes, such as operating room preparation, surgical team preparation, anesthesia process, preoperative patient position adjustment, surgical area disinfection and drape laying, preoperative instrument inventory, postoperative instrument inventory, postoperative patient position adjustment, cleaning and disinfection of the operating room, etc. As for the video clips for calculating the target instrument usage data are in the surgical scene, it is necessary to screen out the non-surgical scenes in the video by scene classification, and only retain the target scene of the intraoperative operation. This scene classification method can be implemented based on deep learning algorithms or scene recognition algorithms. For example, a binary classifier based on a residual network with 18 layers (Resnet18) can be trained in advance to determine whether the image belongs to a non-surgical scene or a surgical scene, and the images of non-surgical scenes are filtered out from the image data of each frame read from the initial video stream and stored in chronological order, and the images of surgical scenes are retained to obtain the frame image to be tested. Of course, the above classifier can also be any other classifier based on a convolutional neural network (CNN), which is not limited here.

[0060] Through the above method, the surgical video is preprocessed based on deep learning, non-surgical scenes are eliminated, and the surgical scenes are retained, providing a more accurate data source and a more efficient analysis process for the subsequent generation of computing instrument usage data.

[0061] Next, target recognition processing is performed on each frame of the image to be tested that has been acquired as mentioned above, and the target instrument in each frame can be identified. In addition, considering that a variety of target instruments are usually required in an operation, such as scalpels, surgical scissors, needle holders, hemostatic forceps, ultrasonic scalpels, etc.; therefore, in order to effectively ensure the accuracy of the generation and recording of instrument usage data, while identifying the target instrument in each frame of the image to be tested, the category information corresponding to the target instrument can also be identified and determined. Specifically, a target recognition algorithm is used for each frame of the surgical scene to output the coordinate information and category information of all target instruments in each frame; the target recognition algorithm can be any target recognition algorithm. The category information of the target instrument in the current frame image that is identified can be expressed as , n is the current frame number, k is the category of the target device; the category information of the target device in the next frame image can be expressed as .

[0062] Step S220, calculating the first position information of the target device in the current frame image in the next frame image, and obtaining the second position information of the target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image.

[0063] Specifically, the calculation process of calculating the first position information of the target device in the current frame image can be implemented as follows: extract the feature data of the target device in the current frame image, match the target area in the next frame image based on the feature data, and determine the position information of the target area. More specifically, first read the images of the current frame and the next frame from the video stream or image sequence. Then use a feature detection algorithm such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF) or corner detection algorithm to identify the key feature points of the target device in the current frame. Generate a descriptor for each feature point so that it can be matched between different images. Based on the feature point pairs matched in the next frame image, calculate the first position information.

[0064] The second position information of the target device in the next frame image can be directly detected in the frame image. For example, when the target is recognized in the next frame image, the second position information of the target device in the frame image can be determined.

[0065] The first position information and / or the second position information may be represented by characteristic coordinates of the target device. For example, the first position information may be represented as ; is the coordinate of the upper left corner of the target device boundary box in the current frame image, is the lower right corner coordinate. The second position information can be expressed as ; is the coordinate of the upper left corner of the target device bounding box in the next frame image, The coordinates of the lower right corner.

[0066] Step S230: determining usage status data of each target device based on the first location information, the second location information and the category information.

[0067] Specifically, the positional relationship between the first position information and the second position information calculated above is compared to determine whether the target devices in the current frame image and the next frame image are the same object. For example, the judgment can be made based on the distance between the position information of the two frames; when it is detected that the distance between the first position information and the second position information is less than a preset distance threshold, it means that the moving distance of the target device in the adjacent frames is short, and it can be considered that there is no situation of replacing the target device, and then the target devices in the two frames are determined to be the same object. At this time, for the target device in the next frame image, there is a usage record before this. On the contrary, if it is determined based on the above detection method that the target devices in the two frames are not the same object, it means that for the target device to be tested in the next frame image, there is no usage record at the current frame moment. Based on the usage record of the target device in the next frame image, the current usage status corresponding to the target device can be determined.

[0068] It should be understood that, for the target device of the current frame image, the above detection method can also be used to determine its usage status. That is, the third position information of the target device of the previous frame image in the current frame image is calculated, and the fourth position information of the target device of the current frame image in the current frame image is obtained, and based on the positional relationship between the third position information and the fourth position information, it is detected whether the target devices in the previous frame image and the current frame image are the same object. In addition, if the current frame image is the first frame image in the above-mentioned frame image to be tested, it can be directly determined that the target device in the current frame image has not been used before. Of course, if the current frame image does not detect the target device, and the target device appears in the next frame image, it means that the position of the target device of the current frame image in the next frame image does not exist, and it can be directly determined that the target device of the next frame is not used in the current frame.

[0069] The above method can be used to detect whether the target device in two adjacent frames of images is the same; in addition, when there are multiple target devices in the frame of image, tracking and matching can also be performed in the above method, which is beneficial to improve the accuracy of automatic recording of the use of the target device.

[0070] In the above steps S220 to S230, each frame image in the frame image to be tested is traversed, and the traversed current frame image and the adjacent frame image are calculated to obtain the position of the target device identified in the current frame in each adjacent frame image in the next frame image, so as to match whether the target device identified in the current frame and the target device identified in the next frame are the same. Based on this judgment, the usage status data of each target device that changes over time can be tracked and recorded, and saved according to the corresponding category information. It should be supplemented that if it is considered that the target device identified in the current frame and the target device identified in the next frame are not the same, that is, the target device is not used in the current frame, or the location information of the target device is not successfully obtained during the target tracking process, then the target device identified in the next frame is considered to be a brand new instance. In this case, the subsequent usage status of this new target device will be re-recorded. Among them, the usage status data of each target device can be expressed in a data structure such as an array or a data table.

[0071] Step S240: Generate usage data of each target device according to the usage status data.

[0072] After obtaining the usage status data of the target device, the usage data such as the usage time data and the usage frequency data of the corresponding target device can be calculated based on a computer program according to whether each target device represented by the usage status data is used in each frame of the image to be tested.

[0073] For example, for each identified target instrument, the timestamp of its first use and the timestamp of its last use can be recorded according to the use status data calculated by the above steps, so as to calculate its total use time. For another example, the change of the state of the target instrument (from unused to used, and then to unused) can be monitored according to the corresponding use status data, so as to automatically count the number of times each target instrument is used during the operation.

[0074] After calculating and generating various aspects of usage data for each target instrument in the above manner, the usage data for each instrument can be integrated with other medical data (such as patient records, surgical results, etc.) to provide a more comprehensive surgical view, which in turn helps to optimize operating room resource allocation, including inventory management and allocation of target instruments.

[0075] In the above-mentioned method for generating instrument usage data, the position of the target instrument of the current frame image in the next frame image between two adjacent frames is determined by target recognition and tracking processing of the frame image to be tested, and based on the judgment of whether the target instrument of the next frame image has been used in the current frame, accurate and convenient adaptive recognition of the usage status of the target instrument in each frame image to be tested that changes over time is achieved, and usage data such as the usage time of each target instrument can be automatically calculated based on the usage status data, so that the present application can not only identify all instruments in the operating room, but also conduct specific analysis of the usage of each target instrument, thereby providing a method for automatically recording instrument usage data based on artificial intelligence and computer vision technology, without the need for manual statistics and recording of instrument usage data, effectively improving the efficiency and accuracy of instrument usage data generation.

[0076] In some embodiments, the above-mentioned calculation of the first position information of the target device in the current frame image in the next frame image may also include the following steps:

[0077] Determine the current frame position information corresponding to the target device in the current frame image; perform segmentation processing on the current frame image according to the current frame position information to obtain a regional frame image. In this step, target recognition can be performed on the current frame image to identify the target frame of the target device in the image, and the target frame coordinates are the current frame position information. The current frame position information can be expressed as ,in is the coordinate of the upper left corner of the target frame of the target device in the current frame, is the lower right corner coordinate. Then, the current frame image is cut according to the detected current frame position information of the target device to obtain the cut region frame image.

[0078] In addition, the above-mentioned regional frame image and the next frame image are input into the trained target tracking model; using the target tracking model, the regional position matching the regional frame image is searched in the next frame image, and the first position information is determined based on the regional position. In this embodiment, the above-mentioned target tracking model can be generated by training such as a fully convolutional twin network (Fully-Convolutional Siamese Networks, SiameseFC) network. Taking the SiameseFC network as an example, the core idea of ​​its algorithm is to use the twin network to learn the target template, and then search for the area that best matches the template in the video sequence to achieve target tracking. Specifically, the SiameseFC network has two inputs, the first input is the image cut out by the coordinate information of the target device detected by the current frame image, that is, the above-mentioned regional frame image, and the second input is the next frame image. The SiameseFC network processes the two input images through two parallel convolutional neural network branches. The two branches have the same network structure and weights. Each branch converts the input image into a feature vector, which is then used to calculate the similarity between the two inputs. For example, the SiameseFC network outputs a response map that shows the similarity between each position in the next frame image and the target device image in the current frame. The peak in the response map usually corresponds to the best matching area, that is, the area searched in the next frame image is the area that best matches the frame image, and the bounding box coordinates of the matched area are output, and finally the first position information mentioned above is obtained.

[0079] Through the above-mentioned embodiment, the movement process of each target device is tracked by using a target tracking algorithm, thereby improving the accuracy of determining the movement position of the target device in each frame image, which is beneficial to improving the accuracy of generating device usage data.

[0080] In some embodiments, the method of determining the usage status data of the target device based on the first location information, the second location information and the category information may further include the following steps:

[0081] Based on the first position information, determine that the target device of the current frame image is in the first target area of ​​the next frame image, and based on the second position information, determine that the target device of the next frame image is in the second target area of ​​the next frame image; determine the degree of overlap between the first target area and the second target area; and determine the usage status data based on the degree of overlap and the category information. Among them, by comparing the degree of overlap between the above two areas, it is detected whether the target device in the current frame image and the target device in the next frame image are the same object, which is conducive to improving the accuracy of detection.

[0082] More specifically, the above-mentioned determination of the overlap degree between the first target area and the second target area further includes the following steps:

[0083] The intersection result of the device area between the first target area and the second target area is calculated, and the overlap degree is determined according to the intersection result of the device area. 目标1 and R 目标2 They represent the area occupied by the first target area and the area occupied by the second target area, respectively, and both areas are represented by the number of pixels. Then the intersection result of the above equipment area is R 目标1 ∩R 目标2 Based on this result, the Intersection over Union (IoU) of the two is calculated as shown in the following formula:

[0084] IoU=(R 目标1 ∩R 目标2 ) / (R 目标1 ∪R 目标2 )

[0085] The above IoU value can be used to indicate the degree of overlap. When it is detected that the IoU is greater than a preset threshold, for example, when the IoU is greater than 0.9, it can be considered that the positions of the target devices in the two adjacent frame images are relatively close, and it can be determined that the target device identified in the current frame and the target device identified in the next frame are the same object.

[0086] Through the above embodiment, by calculating the IoU value between the first position information and the second position information, the overlapping degree of the area where the target device is located in the two frames of images is quantified, which is conducive to improving the accuracy of the target device tracking algorithm.

[0087] In some embodiments, determining the usage status data of each target device based on the first location information, the second location information and the category information includes:

[0088] Based on the first position information and the second position information, detect whether the target device of the next frame image has been used in the current frame, and generate a usage mark of the target device of the next frame image based on the detection result; traverse the frame images to be tested, and determine the usage status data of each target device according to the usage mark and category information corresponding to the target device of each frame image to be tested.

[0089] Specifically, if it is determined based on the comparison result between the first position information and the second position information that the respective target devices of the current frame image and the next frame image are not the same, it can be determined that the target device of the next frame image is not used in the current frame, and the first use mark used to characterize the unused state of the current frame is assigned to the use status of the target device of the next frame image. On the contrary, if it is determined that the respective target devices of the current frame image and the next frame image are the same object, it can be determined that the target device of the next frame image has been used in the current frame, and the second use mark used in the current frame is assigned to the use status of the target device in the next frame image. Traverse each frame image to be tested, and for each frame image to be tested that is traversed, the use status of the target device of the next frame image at the current frame moment is detected by the above method, so as to track the use status of each target device over time. At this time, for each target device, a corresponding use mark is assigned at each frame moment, and finally the use mark and category information can be stored together under the use status data of the corresponding target device.

[0090] Through the above embodiment, by quantitatively marking the usage status of each target device over time, the complexity is simplified, so that the device usage data can be calculated more conveniently and quickly.

[0091] In some embodiments, the above-mentioned determining the usage status data of each target device according to the usage mark and category information corresponding to the target device in each frame image to be detected may also include the following steps:

[0092] If the usage mark indicates that the target device of the current frame image is not used, a corresponding device instance is created based on the target device of the next frame image; and usage status data is determined based on the category information and the device instance.

[0093] Among them, when it is detected that the target device identified in the current frame and the target device identified in the next frame do not belong to the same object, a second use mark is generated based on the detection result, indicating that the target device of the next frame image is not used in the current frame. According to the use mark, or the detection result, the target device identified in the next frame is regarded as a brand new instance. In this case, the subsequent use status of this new target device will be re-recorded. It should be understood that if the coordinate information is not successfully obtained during the above-mentioned target tracking process, for example, the first position information of the target device in the current frame image in the next frame or the current frame position information of the target device in the current frame image is not obtained, there may be a problem of target tracking error at this time, and the target device of the next frame image can also be regarded as a new device instance, and subsequent records are made based on this.

[0094] Taking an array as an example, a set of [0, 1] arrays can be used to represent the usage status of each target device over time, where 0 means that the target device has not been used in the current frame, and 1 means that the target device has been used in the current frame. The array is stored in a dictionary in the form of {"category name": [0, 1 array]}. If the category names are the same, a number is added after the name, for example:

[0095] {“Scalpel 1”:[0,0,0,0,……,0,1,1,1,1,1,1,1,0,0,0,0,0,……,0],

[0096] “Scalpel 2”: [0,0,1,1,……,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,……,0],

[0097] “Hemostatic forceps”: [0,0,0,0,……,0,0,0,0,0,1,1,……,1,1,1,1,1,0,……,0],

[0098] …}.

[0099] In the above example, although scalpel 1 and scalpel 2 belong to the same category of target instruments, in this embodiment, scalpel 1 detected by the above method has not been used before the i-th frame, while scalpel 2 is detected to have been used in the i-th frame. Therefore, scalpel 1 is a newly created instrument instance, and the two need to be distinguished by adding a number after the category name.

[0100] Through the above-mentioned embodiments, the rules for creating new device instances are formulated, thereby improving the statistical method of the usage status data of each target device.

[0101] In some embodiments, the above-mentioned determining the usage status data based on the category information and the device instance may also include the following steps:

[0102] Based on the category information, target devices with the same name among the target devices are determined, and device instances corresponding to the target devices with the same name are merged to obtain usage status data.

[0103] There are two reasons why target instruments have the same name: the first is that the same target instrument is used during the operation; the second is that the tracking algorithm loses tracking in a certain frame. In either case, the arrays of target instruments with the same name need to be merged to accurately count the usage of the target instruments.

[0104] Taking arrays as an example, arrays of target devices with the same name, that is, arrays with numbers in their names, are added together, and values ​​greater than 1 are set to 1. Then, the arrays of target devices with the same name can be merged into one array. The merged form is as follows:

[0105] {“scalpel”: [0,01,1,……,1,1,1,1,1,1,……,1,0,0,0,0,0,……,0],

[0106] “Hemostatic forceps”: [0,0,0,0,……,0,0,0,0,0,1,1,……,1,1,1,1,1,0,……,0],

[0107] …}.

[0108] Through the above embodiment, by merging the usage status data of target devices with the same name, the problem of affecting the calculation of target device usage data caused by the loss of tracking of a certain frame is effectively avoided, and the statistical results of the usage status data are further simplified, which is conducive to improving the accuracy of the generation of device usage data.

[0109] In the above embodiment, a merge optimization method for the usage status data of the target device is provided. In addition, in another embodiment, the usage status data can also be subjected to discrete point smoothing processing to improve the accuracy of subsequent usage data calculation. The usage status data is stored in an array form; the usage status array can be smoothed by median filtering to process discrete points; the window length of the median filter is set to 1 second, and the corresponding median filter window size is also determined by the video frame rate, thereby obtaining an optimized usage status array.

[0110] In some embodiments, the above-mentioned step of generating usage data of each target device according to the usage status data may further include the following steps:

[0111] According to the usage status data, the number of frames in which the corresponding target device appears in the frame image to be measured is calculated; based on the number of frames, the usage time of the target device is calculated, and according to the usage time and the acquired total usage time, the proportion of the total usage time of the target device is calculated; according to the usage status data, the number of consecutive appearances of the corresponding target device in the frame image to be measured is determined; according to the number of consecutive appearances, the number of uses of the target device is determined; wherein the usage data includes the usage time, the proportion of the total usage time and the number of uses.

[0112] Specifically, the above usage time data may include the usage time of the target device, the percentage of total usage time, the percentage of the maximum usage time of each use, the percentage of the minimum usage time of each use, and the percentage of the average usage time of each use. The usage time can be calculated by detecting the number of frames in the image to be tested in the usage status data corresponding to a certain target device, and the usage time of the target device can be calculated based on the sum of the number of frames and the video frame frequency. For example, if the video frame rate FPS=60, if the target device appears in all 60 frames of images, the usage time is calculated to be 1s; if it appears in all 600 frames of images, the usage time is calculated to be 10s.

[0113] The above-mentioned usage ratios refer to the ratios to the total duration of the operation. For example, the total usage time ratio is calculated based on the ratio between the usage time of the target instrument and the total usage time. For another example, the maximum duration ratio of each use can be calculated based on the maximum value of the number of consecutive frames in each consecutive number of occurrences, and then the maximum duration ratio is calculated; specifically, the number of consecutive frames in the usage status data corresponding to a certain target instrument is counted; for example, for the target instrument, the 10th to 20th frames appear continuously, and the 30th to 45th frames appear continuously, then the number of consecutive frames is 2; then the maximum value in each consecutive number of frames is counted, for example, the 30th to 45th frames appear continuously this time is the maximum value of the statistics, that is, 15 frames appear continuously this time, then the maximum duration of the target instrument is calculated based on the maximum value and the video frame rate, and then the maximum duration ratio of each use is calculated based on the ratio between the maximum duration of use and the total duration of the operation. It should be understood that for other types of duration ratio calculations, they can also be implemented by referring to the above calculation method, which will not be repeated here.

[0114] In addition, the above-mentioned number of uses can be calculated by the number of times the target device appears continuously in at least one frame of the image to be tested without interruption, that is, the above-mentioned number of consecutive appearances; for example, if it is counted that it appears continuously from the 10th to the 20th frame and from the 30th to the 45th frame, then the number of consecutive appearances of the target device is 2, that is, the number of uses is 2.

[0115] Taking the use status data in the form of an array as an example, the calculation formulas for various types of use data can be expressed by the following formulas:

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122] In the above formula, An array representing whether a target instrument is used or not at each frame time; Sum() is a summation function. T is the total duration of the surgery. Len() is a function for finding the length of the array. ones The array stores The number of 1s in each segment when the continuous value is 1 can be obtained by counting. For example, if Arr=[1,1,1,1,0,0,1,1,0,1,1,1,0,0,0], then Arr ones =[4,2,4]; where 1 indicates that the target device is used in this frame, and 0 indicates that the target device is not used in this frame. c indicates the number of times the target device is used, Max() is the maximum value function, Min() is the minimum value function, and Mean() is the average value function. FPS indicates the video frame rate.

[0123] Through the above embodiments, a complete calculation method for generating various types of usage data based on usage status data is provided, and users can choose to set the calculation of some or all types of usage data according to actual conditions, thereby expanding the application scenarios.

[0124] The embodiments of the present application are described and illustrated below through preferred embodiments. Figure 3 is a flow chart of another method for generating device usage data according to an embodiment of the present application. Figure 3 As shown, the process includes the following steps:

[0125] Step S301, video preprocessing: Specifically, each frame of the collected initial video stream containing the preoperative scene, the intraoperative scene and the postoperative scene is read out, saved as a picture file in chronological order, and the frame rate of the video is marked as FPS.

[0126] Step S302, classify the scenes in the video. The video clip for calculating the target instrument usage data is in a surgical scene, so it is necessary to filter out the non-surgical scenes in the video by scene classification, and only retain the frame images to be tested of the surgical scene.

[0127] Step S303, identify and track the target instrument to obtain an array of target instrument usage status. Specifically includes the following process: use the target recognition algorithm for each frame image of the surgical scene, and output the current frame position information and category information of all target instruments in each frame image. For each two adjacent frames of images, use the target tracking algorithm to output the first position information of the target instrument identified in the current frame in the next frame image. According to the first position information and the second position information of the target instrument in the next frame image in the next frame image, match whether the target instrument identified in the current frame and the target instrument identified in the next frame are the same, and obtain the target instrument usage status array. In the above manner, the target recognition and target tracking algorithms are used to track the movement process of each target instrument to ensure that the target instrument counted is the same, so that the statistical data is more accurate.

[0128] Step S304, target device usage status array processing. The array processing may include merging arrays of target devices with the same name and array discrete point processing to further optimize the usage status array.

[0129] Step S305, calculating the equipment usage data based on the usage status array optimized by the above step S304.

[0130] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0131] This embodiment also provides a device for generating instrument usage data, which is used to implement the above embodiments and preferred implementations, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0132] Figure 4 is a structural block diagram of a device for generating device usage data according to an embodiment of the present application, such as Figure 4As shown, the device includes: an identification module 41, used to obtain the frame image to be tested, and determine the category information of the target device identified from each frame image to be tested; a position tracking module 42, used to calculate the first position information of the target device in the current frame image in the next frame image, and obtain the second position information of the target device in the next frame image in the next frame image; wherein the frame image to be tested includes the current frame image and the next frame image; a usage status determination module 43, used to determine the usage status data of each target device based on the first position information, the second position information and the category information; a generation module 44, used to generate the usage data of each target device according to the usage status data.

[0133] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination. For specific examples in this embodiment, reference can be made to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0134] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0135] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0136] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0137] S1, obtaining a frame image to be tested, and determining the category information of the target device identified from each frame image to be tested.

[0138] S2, calculating the first position information of the target device in the current frame image in the next frame image, and obtaining the second position information of the target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image.

[0139] S3, determining usage status data of each target device based on the first location information, the second location information and the category information.

[0140] S4, generating usage data of each target device based on the usage status data.

[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0142] In addition, in combination with the method for generating device usage data in the above embodiments, the present application embodiment may provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, any of the methods for generating device usage data in the above embodiments is implemented.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0144] Those skilled in the art should understand that the technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating device usage data, characterized in that: The method comprises: Acquire the frame images to be tested, and determine the category information of the target device identified from each of the frame images to be tested; Calculating first position information of a target device in a current frame image in a next frame image, and obtaining second position information of a target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image; Determining usage status data of each target device over time based on the first location information, the second location information, and the category information, including: Based on the first position information and the second position information, detecting whether the target device of the next frame image has been used in the current frame, and generating a use mark of the target device of the next frame image based on the detection result; Traversing the frame images to be tested, and determining the usage status data of each target device according to the usage mark and the category information corresponding to the target device of each frame image to be tested, including: If the use mark indicates that the target device of the current frame image is not used, creating a corresponding device instance based on the target device of the next frame image; Based on the category information, determine target devices with the same name among the target devices, and merge the device instances corresponding to the target devices with the same name to obtain the usage status data; Among them, for the usage status of each target device that changes over time, a set of [0, 1] arrays are used to represent it, 0 means that the target device has not been used in the current frame, and 1 means that the target device has been used in the current frame, and the array is stored in the dictionary in the form of {"category name": [0, 1]}; Based on the usage status data, usage data of each of the target devices is generated.

2. The generation method according to claim 1, characterized in that: The step of calculating first position information of a target device in a current frame image in a next frame image includes: Determine the current frame position information corresponding to the target device in the current frame image; Performing a segmentation process on the current frame image according to the current frame position information to obtain a regional frame image; The regional frame image and the next frame image are input into a trained target tracking model; using the target tracking model, a regional position matching the regional frame image is searched in the next frame image, and the first position information is determined based on the regional position.

3. The generation method according to claim 1, characterized in that: The determining the usage status data of the target device based on the first location information, the second location information and the category information includes: Based on the first position information, determining that the target device of the current frame image is in a first target area of ​​the next frame image, and based on the second position information, determining that the target device of the next frame image is in a second target area of ​​the next frame image; Determine the degree of overlap between the first target area and the second target area; and determine the usage status data based on the degree of overlap and the category information.

4. The generation method according to claim 3, characterized in that: The determining the degree of overlap between the first target area and the second target area comprises: The intersection result of the device area between the first target area and the second target area is calculated, and the overlap degree is determined according to the intersection result of the device area.

5. The generation method according to any one of claims 1 to 4, characterized in that: The step of acquiring the frame image to be tested comprises: Get the initial video stream; Scene recognition is performed on the initial video stream to determine a target frame image in the initial video stream that is in a target scene, and the frame image to be tested is determined based on the target frame image.

6. The generation method according to any one of claims 1 to 4, characterized in that: Generating usage data of each target device according to the usage status data includes: According to the usage status data, calculating the number of frames in which the corresponding target device appears in the frame image to be tested; Calculating the usage time of the target device based on the number of occurrence frames, and calculating the total usage time ratio of the target device according to the usage time and the acquired total usage time; Determine the number of consecutive appearances of the corresponding target device in the frame image to be tested according to the usage status data; determine the number of times the target device is used according to the consecutive number of appearances; The usage data includes the usage time, the percentage of the total usage time and the number of times of usage.

7. The generation method according to any one of claims 1 to 4, characterized in that: The usage status data is expressed in the form of an array.

8. A device for generating equipment usage data, characterized in that: include: An identification module, used to obtain the frame image to be tested and determine the category information of the target device identified from each of the frame images to be tested; A position tracking module, used to calculate the first position information of the target device in the current frame image in the next frame image, and obtain the second position information of the target device in the next frame image in the next frame image; wherein the frame image to be measured includes the current frame image and the next frame image; A usage status determination module, configured to determine usage status data of each target device that changes over time based on the first location information, the second location information and the category information; The usage status determination module is further used to detect whether the target device of the next frame image has been used in the current frame based on the first position information and the second position information, and generate a usage mark of the target device of the next frame image based on the detection result; The usage status determination module is further used to traverse the frame images to be tested, and determine the usage status data of each target device according to the usage mark and the category information corresponding to the target device of each frame image to be tested; The usage status determination module is further configured to create a corresponding device instance based on the target device of the next frame image if the usage mark indicates that the target device of the current frame image is not used; The usage status determination module is further used to determine target devices with the same name among the target devices based on the category information, and merge the device instances corresponding to the target devices with the same name to obtain the usage status data; Among them, for the usage status of each target device that changes over time, a set of [0, 1] arrays are used to represent it, 0 means that the target device has not been used in the current frame, and 1 means that the target device has been used in the current frame, and the array is stored in the dictionary in the form of {"category name": [0, 1]}; A generating module is used to generate usage data of each target device according to the usage status data.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method for generating instrument usage data according to any one of claims 1 to 7 when running.

Images