Video stream processing method and apparatus, computer device, and storage medium

Abstract

The application relates to a video stream method, device, computer equipment and storage medium, information of a video frame in a video stream is acquired, the information of the video frame comprises sub-region masks of multiple sub-regions; the sub-region masks are combined, a mask graph of a candidate landmark part is generated according to the sub-region masks and the combined sub-region masks; a heat map is generated according to the mask graph, and a combined graph is obtained by combining the mask graph and the heat map, features of the candidate landmark part are extracted according to the combined graph; the features of the candidate landmark part are compared with features in a preset library, and actual landmark parts of the video frame are recognized according to a comparison result. The method can improve the recognition accuracy of the landmark parts and solve the problem of low accuracy when the landmark parts are recognized through an endoscopic examination video.

Description

Technical Field

[0001] This application relates to the field of video image processing technology, and in particular to a video streaming method, apparatus, computer device, and storage medium. Background Technology

[0002] Endoscopy, as an important tool for diagnosis and treatment in modern medicine, is widely used in the examination of various parts of the human body, such as the gastrointestinal tract, nasopharynx, bladder, and bronchi. Doctors can use endoscopes to examine the lesions in various parts of the patient in detail and make better diagnoses. The most critical part of endoscopic examination is to examine various landmarks to ensure that no landmarks are missed. However, in actual examinations, due to the complexity of the internal structure of the human body and the large number of landmarks required for different parts, repeated examinations may be necessary, which may cause discomfort to the patient. It may also affect the quality of the examination if some landmarks are not captured.

[0003] In related technologies, a topological structure is established using over a dozen landmark sites in the nasopharynx and larynx, and key nodes are extracted to achieve landmark site extraction. These landmark sites are then identified using rectangular bounding boxes. However, during actual examinations, many sites change shape due to the patient's breathing and slight body movements. This alters the topological structure of the key points, leading to inaccurate detection. Furthermore, many landmark sites lack a clear structure, making key point extraction impossible. Moreover, during endoscopy, many sites are adhered together; rectangular bounding box recognition cannot eliminate interference from these adhesions, further reducing accuracy.

[0004] There is currently no effective solution to the problem of low accuracy in identifying landmarks using endoscopic video examination in related technologies. Summary of the Invention

[0005] Therefore, it is necessary to provide a video stream method, apparatus, computer equipment, and storage medium that can improve the recognition accuracy of marker parts in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a video stream processing method, the method comprising:

[0007] Obtain information about video frames in a video stream, wherein the information about video frames includes sub-region masks of multiple sub-regions;

[0008] Combine the sub-region masks, and generate a mask image of the candidate marker locations based on the sub-region masks and the combined sub-region masks;

[0009] A heatmap is generated based on the mask image, and a combined image is obtained by combining the mask image and the heatmap. Features of the candidate marker regions are extracted based on the combined image.

[0010] Compare the features of the candidate landmark with features in a preset library, and identify an actual landmark of the video frame according to a comparison result.

[0011] In one of the embodiments, information of a video frame in a video stream is acquired, including:

[0012] A video stream generated during an endoscopy is received, and a video frame in the video stream is extracted;

[0013] The video frame is segmented, and information of the video frame is acquired according to a segmentation result.

[0014] In one of the embodiments, a mask map of a candidate landmark is generated according to the sub-region mask and the combined sub-region mask, including:

[0015] A first sub-region mask of a first sub-region in the video frame and a second sub-region mask of a second sub-region in the video frame are acquired, wherein a distance between the first sub-region and the second sub-region is within a preset range;

[0016] When the distance between the first sub-region and the second sub-region is within a first preset range, the mask map is generated according to the first sub-region mask;

[0017] When the distance between the first sub-region and the second sub-region is within a second preset range, the mask map is generated according to the first sub-region mask and the combined sub-region mask.

[0018] In one of the embodiments, a mask map of a candidate landmark is generated according to the sub-region mask and the combined sub-region mask, including:

[0019] A candidate landmark is determined according to the sub-region mask and the combined sub-region mask;

[0020] The candidate landmark and a first preset value are acquired, and the candidate landmark is assigned a value according to the first preset value;

[0021] A non-candidate landmark other than the candidate landmark in the video frame and a second preset value are acquired, and the non-candidate landmark is assigned a value according to the second preset value;

[0022] The mask map is generated according to a result of the assignment.

[0023] In one of the embodiments, combining the mask map and the heat map includes:

[0024] Channels of the mask map and channels of the heat map are merged according to a preset weight.

[0025] In one embodiment, the method comprises:

[0026] obtaining a first sub-region code of a first sub-region and a second sub-region code of a second sub-region;

[0027] combining the second sub-region code based on the first sub-region code, and obtaining a mask queue corresponding to the first sub-region according to a combination result and the first sub-region code;

[0028] numbering the first sub-region, and adding the mask queue corresponding to the first sub-region into a comparison queue according to a numbering order of the first sub-region;

[0029] determining a comparison order of the candidate landmark part through the comparison queue.

[0030] In one embodiment, before comparing a feature of the candidate landmark part with a feature in a preset library, the method further comprises:

[0031] obtaining a corresponding landmark part database according to a type of the video stream;

[0032] obtaining a landmark part map in the landmark part database, generating a mask map of the landmark part map and a heat map of the landmark part map, and obtaining a combined map of the landmark part map by combining the mask map of the landmark part map and the heat map of the landmark part map;

[0033] extracting a feature of the landmark part map according to the combined map of the landmark part map to obtain the feature of the landmark part map, and obtaining the preset library.

[0034] In a second aspect, the application further provides a video stream processing device, which comprises a checking module, a processing module and a display module, wherein the checking module is connected with the processing module, and the processing module is connected with the display module; and wherein,

[0035] the checking module is configured to obtain a video stream;

[0036] the processing module is configured to implement the steps of the method according to any one of the first aspect;

[0037] the display module is configured to display a processing result of the processing module.

[0038] In a third aspect, the application further provides a computer device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method according to any one of the first aspect when executing the computer program.

[0039] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the steps of the method of any one of the first aspect.

[0040] The video stream method, device, computer device and storage medium described above obtain information of a video frame in a video stream, the information of the video frame including sub-region masks of a plurality of sub-regions; the sub-region masks are combined, a mask map of a candidate landmark is generated according to the sub-region masks and the combined sub-region masks; a heat map is generated according to the mask map, and features of the candidate landmark are extracted according to the combined mask map and heat map in combination with the mask map and the heat map; the features of the candidate landmark are compared with features in a preset library, and an actual landmark of the video frame is recognized according to a comparison result, thereby improving the recognition accuracy of the landmark and solving the problem of low accuracy in recognizing the landmark from an endoscopic examination video. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 A flowchart of a video stream processing method in an embodiment of the present application;

[0042] Figure 2 A flowchart of landmark recognition in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a mask-based retrieval method in an embodiment of the present application;

[0044] Figure 4 A video stream processing device in an embodiment of the present application;

[0045] Figure 5 An internal structure diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] In one embodiment, as shown in Figure 1 A flowchart of a video stream processing method is provided, and this embodiment is exemplified by applying the method to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. In this embodiment, the method includes the following steps:

[0048] At step 101, information of a video frame in a video stream is acquired, the information of the video frame comprising sub-region masks of a plurality of sub-regions.

[0049] The video stream comprises a video transmitted in real time during an endoscopy. All regions included in the video frame are identified and segmented to obtain the sub-regions and the masks of the sub-regions. Optionally, the video stream is extracted and the video frame is obtained, and the video frame is segmented to divide the video frame into a plurality of different and non-overlapping sub-regions and the masks of the sub-regions.

[0050] At step 102, the sub-region masks are combined, and a mask map of a candidate landmark is generated according to the sub-region masks and the combined sub-region masks.

[0051] The landmark comprises a region in the video frame that has a significant feature in terms of visual perception. A complete landmark can be a single sub-region or a plurality of sub-regions combined. Therefore, the sub-region masks are combined to obtain the sub-region masks and the combined sub-region masks. The sub-regions corresponding to the sub-region masks or the sub-regions combined according to the combined sub-region masks are candidate landmarks.

[0052] At step 103, a heat map is generated according to the mask map, and a combined map is obtained by combining the mask map and the heat map, and features of the candidate landmark are extracted according to the combined map.

[0053] The mask map is subjected to saliency detection, and a heat map is generated according to the saliency features in the mask map. The heat map is used to determine a high-response region with high saliency in the mask map. Optionally, the mask map and the heat map are combined by image superposition, and features of the candidate landmark are extracted based on the mask map and the heat map.

[0054] At step 104, the features of the candidate landmark are compared with features in a preset library, and an actual landmark of the video frame is identified according to a comparison result.

[0055] The preset library comprises features of preset landmarks that can be involved in any endoscopy. Identifying the actual landmark of the video frame comprises: identifying whether the actual landmark exists in the video frame, and determining a category of the actual landmark.

[0056] Optionally, according to the type of the video stream, a preset library corresponding to the type of the video stream is selected, and the features of the candidate landmark part are compared with the features in the preset library. All candidate landmark part features corresponding to the video frame are compared, and in the case of successful comparison, the category of the actual landmark part of the video frame is determined; in the case of failed comparison, it is determined that there is no actual landmark part in the video frame. By comparing the method of retrieving whether the actual landmark part is included in the video frame, only by switching the type of the preset library, the identification of the landmark part of different types of endoscopy and different examination parts can be realized, without training multiple endoscopic models, and the applicability is high.

[0057] In the above video stream processing method, a sub-region in a video frame of a video stream is obtained, and a candidate landmark part is obtained according to the sub-region and the combination of the sub-region. In the case of shape change of the patient's body part in endoscopy, the candidate landmark part can also be accurately obtained. On the basis of combining the mask map and the heat map, the features of the candidate landmark part are extracted, the false positives caused by the adhesion of the endoscopy area are reduced, the accuracy of the identification is improved, and the problem of low accuracy in identifying the landmark part through the endoscopy video is solved.

[0058] In one of the embodiments, information of a video frame in a video stream is obtained, including: receiving a video stream generated during endoscopy, and extracting a video frame in the video stream; segmenting the video frame, and obtaining information of the video frame according to the segmentation result.

[0059] In time sequence, video frames in a video stream are extracted, and the extracted video frames are segmented respectively. In which, a panoramic segmentation model is trained through a deep learning network such as VGG (Visual Geometry Group), Resnet (Residual Network), Unet (U-shaped network), and an endoscopic video corresponding to different endoscopy methods. The panoramic segmentation model identifies and segments all regions in the video frame to obtain information of the video frame, and the information of the video frame includes multiple sub-regions, sub-region masks, and numbers of the sub-regions.

[0060] In one of the embodiments, a mask map of a candidate landmark part is generated according to a sub-region mask and a combined sub-region mask, including: obtaining a first sub-region mask of a first sub-region in a video frame, and a second sub-region mask of a second sub-region in the video frame, wherein the distance between the first sub-region and the second sub-region is within a preset range; when the distance between the first sub-region and the second sub-region is within a first preset range, generating the mask map according to the first sub-region mask; and when the distance between the first sub-region and the second sub-region is within a second preset range, generating the mask map according to the first sub-region mask and the combined sub-region mask.

[0061] The first sub-region is any sub-region in the video frame. According to the endoscope part corresponding to the video stream, a corresponding preset range is determined. For example, the internal region of the first sub-region corresponds to a region distance of 0, the region distance of the region adjacent to the first sub-region is 1, and so on. At this time, the first preset range corresponds to a region distance of 0, and the mask graph can be directly generated according to the first sub-region mask. The second preset range can be equal to the region distance 1 or greater than the region distance 1. Taking the second preset range corresponding to the region distance 1 as an example: the second sub-region adjacent to the first sub-region A and the second sub-region mask are obtained. If there are N second sub-regions adjacent to the first sub-region A, then there are N!+1 ways of combining the sub-region masks of the candidate marker parts, and then there are N!+2 ways of combining the masks, plus the first sub-region A itself.

[0062] The first sub-region encoding of the first sub-region and the second sub-region encoding of the second sub-region are obtained; the second sub-region encoding is combined on the basis of the first sub-region encoding, and a mask queue corresponding to the first sub-region is obtained according to the combination result and the first sub-region encoding; the first sub-region is numbered, and the mask queue corresponding to the first sub-region is added to the comparison queue according to the numbering order of the first sub-region; and the comparison order of the candidate marker parts is determined through the comparison queue.

[0063] For example, when the first sub-region is region A and the region distance is 1, the mask queue corresponding to the first sub-region A is obtained, the mask queue includes N!+2 mask combinations, and each mask combination in the queue corresponds to a candidate marker part. According to the numbering order of the first sub-region, all mask combinations corresponding to the first sub-region in the video frame are integrated together, and the comparison queue can be obtained. The comparison queue includes the masks of all candidate marker parts in the video frame. The numbering is used to ensure that there is no missing mask in the comparison queue.

[0064] According to the sub-region mask and the combined sub-region mask, the candidate marker part is determined; the candidate marker part and the first preset value are obtained, and the candidate marker part is valued according to the first preset value; the non-candidate marker part in the video frame except the candidate marker part and the second preset value are obtained, and the non-candidate marker part is valued according to the second preset value; and the mask graph is generated according to the valuation result.

[0065] The masks in the comparison queue include the sub-region mask and the combined sub-region mask, and each mask corresponds to a candidate marker part. Optionally, the first preset value is 1 and the second preset value is 0, the candidate marker part is assigned 1, and the non-candidate marker part is assigned 0, and the mask graph of the candidate marker part is obtained. The mask graph is used to avoid the interference of the non-candidate marker part in feature extraction. In order to improve the recognition accuracy, the mask graph is subjected to saliency detection, and a heat map is obtained according to the detected saliency features.

[0066] Optionally, edge detection (e.g., Sobel feature detection) is performed on the mask image to obtain an edge detection grayscale image. Feature points are extracted from the mask image (e.g., ORB feature extraction). Based on the extracted feature points, the number of feature points is obtained. The number of feature points near each pixel is counted to obtain a feature count map. A heatmap is obtained based on the edge detection grayscale image and the number of feature points. In the heatmap, the larger the value corresponding to a region, the more features that region has and the more important it is, thus giving it a greater weight in subsequent feature comparisons. The heatmap can further avoid interference from non-candidate marker regions and increase attention to high-response regions.

[0067] Combining the mask image and heatmap includes: merging the channels of the mask image and the heatmap according to preset weights. Optionally, the channels of the mask image and heatmap are superimposed according to preset weights using the `concat` function. A combined image is obtained from the combined mask image and heatmap, and features of candidate marker regions are extracted from the combined image. Optionally, a combined image is obtained from the mask image and heatmap. The combined image is fed into a deep network (such as a ResNet residual network) to extract a feature vector corresponding to the combined image. The feature vector is used as the feature of the candidate marker region.

[0068] In one embodiment, before comparing the features of candidate marker regions with features in a preset library, the method further includes: obtaining a corresponding marker region database based on the type of the video stream; obtaining marker region images from the marker region database, generating a mask image and a heatmap of the marker region images, and combining the mask image and the heatmap of the marker region images; extracting features of the marker region images based on the combined mask image and the heatmap of the marker region images to obtain a preset library.

[0069] The landmark site database includes base databases of landmark sites for different types of endoscopy, each storing corresponding landmark site images. Based on the video stream type, the required landmark site database is selected, and features of the landmark site images are extracted. The features of candidate landmark sites and the features of the landmark site images are compared separately to determine whether a corresponding landmark site exists in the comparison queue, and to ascertain the category of the actual landmark site.

[0070] In one embodiment, such as Figure 2 As shown, Figure 2 This is a flowchart of a landmark recognition process in this embodiment. The input is an endoscopic video stream, which is continuously analyzed in real-time as the doctor performs the examination. Landmark areas are then identified and output.

[0071] Endoscopic panoramic segmentation, obtain sub-region mask. Among them, the specific part of the sub-region is not distinguished. Through deep learning technology, including but not limited to adopting VGG, Resnet, Unet and other networks to perform panoramic segmentation on endoscopic video frames, different regions of the video frames are divided out and numbered.

[0072] Establish a part candidate mask queue. In addition to each sub-region obtained by panoramic segmentation, the sub-region needs to be combined to some extent. After sub-region combination, the mask queue corresponding to each sub-region mask can be obtained, and each mask in the mask queue may be a landmark part. Integrating all mask queues together forms a landmark part candidate mask queue, that is, a comparison queue. Among them, the comparison queue is dynamically in and out, and the mask after the comparison is out of the queue, and the sub-region mask extracted in the new video frame is entered into the queue in real time. When the comparison queue is empty, the entire endoscopic examination is completed.

[0073] According to different endoscopes, different landmark part databases are selected, and feature extraction and retrieval are based on mask and heat fusion. Figure 3 is a mask-based retrieval method schematic diagram in the embodiment, as shown in Figure 3 , the mask in the candidate landmark part queue is obtained, and the candidate landmark part corresponding to the mask is obtained. For each candidate landmark part, all the non-candidate landmark parts around it are assigned to 0 to obtain a mask graph, which can avoid false positives and interference from the surrounding area during feature calculation. In addition, in order to ensure the effect, the mask graph is subjected to saliency analysis to obtain a feature heat map, which further focuses on the high response area and ignores the unimportant area. Based on the two, the features of the candidate landmark part are obtained.

[0074] According to the endoscope category corresponding to the video stream, the corresponding landmark part database, that is, the landmark part database, is selected. At the same time, the landmark part graph in the landmark part database also realizes feature extraction based on mask and heat map to obtain the features of the landmark part. The features of the candidate landmark part and the features of the landmark part are compared to realize the retrieval of the landmark part. The video stream retrieval method in the embodiment is not limited to the endoscope category, and any endoscope can be used. In actual use, if the corresponding type of the video stream is fixed, the features of the labeled part graph of the corresponding landmark part database can be directly extracted, and the features of the labeled part graph are stored to obtain a preset library.

[0075] Accurate part recognition. According to the comparison result, it is judged whether the corresponding landmark part exists in the comparison queue and the accurate landmark part category.

[0076] Real-time output of snapshot position. In the case of judging whether the corresponding landmark part exists in the comparison queue, the video frame corresponding to the candidate region is obtained, and the video frame is output to the terminal.

[0077] The landmark site recognition method in the embodiment is based on mask retrieval on the basis of panoramic segmentation, and is compatible with the differences of different endoscopes and different sites. In actual use, multiple different endoscope model models do not need to be trained, and different landmark site libraries only need to be selected for different endoscope categories to be directly applicable, which is easy to use. At the same time, the retrieval based on the mask and heat can greatly improve the adhesion false alarm of the endoscope region, and significantly improve the accuracy of landmark site recognition.

[0078] It should be understood that, although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiments of the present application also provide a video stream processing device for implementing the video stream processing method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more video stream processing device embodiments provided below can refer to the limitations of the video stream processing method described above, and will not be repeated here.

[0080] In one embodiment, as shown in Figure 4 A video stream processing device is provided, which includes an inspection module, a processing module and a display module, the inspection module and the processing module are connected, and the processing module and the display module are connected; wherein,

[0081] The inspection module is configured to obtain a video stream;

[0082] The processing module is configured to implement the steps of any one of the embodiments described above;

[0083] The display module is configured to display the processing result of the processing module.

[0084] Each module in the above video stream processing device can be realized by software, hardware and their combinations in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in the form of software, so as to call and execute the operations corresponding to each module by the processor.

[0085] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 5 The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store preset library data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a video stream processing method.

[0086] Those skilled in the art can understand that Figure 5 The structure shown in FIG. 1 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0087] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any one of the above embodiments.

[0088] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of any one of the above embodiments.

[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0090] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0091] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A video stream processing method, characterized in that, The method includes: Obtain information about video frames in a video stream, wherein the information about video frames includes sub-region masks of multiple sub-regions; Combine the sub-region masks, and generate a mask image of the candidate marker locations based on the sub-region masks and the combined sub-region masks; A heatmap is generated based on the mask image, and a combined image is obtained by combining the mask image and the heatmap. Features of the candidate marker regions are extracted based on the combined image. Compare the features of the candidate marker regions with the features in a preset library, and identify the actual marker regions of the video frame based on the comparison results; The process of generating a mask image for candidate marker regions based on the sub-region mask and the combined sub-region mask includes: obtaining a first sub-region mask for a first sub-region in the video frame and a second sub-region mask for a second sub-region in the video frame, wherein the distance between the first sub-region and the second sub-region is within a preset range; when the distance between the first sub-region and the second sub-region is within a first preset range, generating the mask image based on the first sub-region mask; when the distance between the first sub-region and the second sub-region is within a second preset range, generating the mask image based on the first sub-region mask and the combined sub-region mask.

2. The method according to claim 1, characterized in that, Obtain information about video frames in the video stream, including: Receive the video stream generated during an endoscopic examination and extract video frames from the video stream; The video frame is segmented, and information about the video frame is obtained based on the segmentation result.

3. The method according to claim 1, characterized in that, Based on the sub-region mask and the combined sub-region mask, a mask image for candidate marker regions is generated, including: Based on the sub-region mask and the combined sub-region mask, the candidate flag locations are determined; Obtain the candidate marker location and a first preset value, and assign a value to the candidate marker location based on the first preset value; Obtain the non-candidate marker regions other than the candidate marker regions in the video frame and the second preset value, and assign a value to the non-candidate marker regions according to the second preset value; The mask image is generated based on the result of the assignment.

4. The method according to claim 1, characterized in that, The combination of the mask image and the heat map includes: The channels of the mask image and the channels of the heat map are merged according to preset weights.

5. The method according to claim 1, characterized in that, include: Obtain the first sub-region code of the first sub-region and the second sub-region code of the second sub-region; Based on the first sub-region encoding, the second sub-region encoding is combined, and according to the combination result and the first sub-region encoding, a mask queue corresponding to the first sub-region is obtained; The first sub-region is numbered, and the mask queue corresponding to the first sub-region is added to the comparison queue according to the numbering order of the first sub-region; The comparison order of the candidate marker regions is determined by the comparison queue.

6. The method according to claim 1, characterized in that, Before comparing the features of the candidate marker regions with features in a preset library, the method further includes: Based on the type of the video stream, obtain the corresponding marker region database; Obtain the marker part image from the marker part database, generate a mask image and a heat map of the marker part image, and combine the mask image and the heat map of the marker part image to obtain a combined image of the marker part image. Based on the combined image of the marked part image, the features of the marked part image are extracted to obtain the preset library.

7. A video stream processing device, characterized in that, The device includes an inspection module, a processing module, and a display module. The inspection module and the processing module are connected, and the processing module and the display module are connected. The inspection module retrieves the video stream. A processing module is configured to implement the steps of the method according to any one of claims 1 to 6; The display module displays the processing results of the processing module.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images