An intelligent clipping application method and device through picture recognition, equipment and medium

Abstract

The application relates to an intelligent clipping application method through picture recognition, which comprises the following steps: acquiring medical image video data to be clipped; performing grouped image recognition on the medical image video data to be clipped to obtain a key frame index; and performing clipping on the medical image video data to be clipped based on the key frame index. The application does not require a clipping personnel with medical knowledge to perform clipping, and the clipping personnel can complete the video clipping operation without watching the whole content of the video or repeatedly watching the video for multiple times, so that the efficiency of the video clipping is improved. The application also relates to an intelligent clipping application device through picture recognition, a storage medium and equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to an intelligent clipping application method and device based on picture recognition, equipment and medium. BACKGROUND

[0002] Video clipping technology is a video processing method of synthesizing a clip video by clipping a to-be-clipped object such as a static image or a dynamic video. The video clipping technology is commonly applied to short video production and video highlight scenes.

[0003] Traditional video clipping technology requires a clipping personnel to watch the entire video content, and then manually clips the required video segment. This video clipping method requires a large amount of energy and time of the video clipping personnel, and is inefficient, especially when the video to be clipped is a medical image video. The medical image video does not have subtitle and voice, dialogue and other information, and therefore requires a clipping personnel with medical knowledge to be familiar with the video content before reasonable clipping, which reduces the video clipping efficiency. SUMMARY

[0004] The present application solves the technical problems of the prior art, and provides an intelligent clipping application method and device based on picture recognition, equipment and medium.

[0005] The technical solutions of the present application for solving the above technical problems are as follows:

[0006] An intelligent clipping application method based on picture recognition comprises the following steps:

[0007] Obtaining medical image video data to be clipped;

[0008] Performing grouped image recognition on the medical image video data to be clipped to obtain a key frame index;

[0009] Clipping the medical image video data to be clipped based on the key frame index.

[0010] The present application provides an intelligent clipping application method based on picture recognition, which comprises the following steps: obtaining medical image video data to be clipped; performing grouped image recognition on the medical image video data to be clipped to obtain a key frame index; and clipping the medical image video data to be clipped based on the key frame index. The present application does not require a clipping personnel with medical knowledge to perform clipping, and the clipping personnel only needs to watch the entire video content or repeatedly watch the video several times to complete the video clipping operation, thereby improving the video clipping efficiency.

[0011] On the basis of the above technical solutions, the present application can be further improved as follows.

[0012] Furthermore, the step of performing grouped image recognition on the medical image video data to be edited to obtain keyframe indexes specifically includes:

[0013] Random noise in the medical image video data is eliminated using the k-nearest neighbor average method.

[0014] Image enhancement processing is performed on the medical image video data using histogram equalization and logarithmic transformation.

[0015] The medical image video data is segmented into segments corresponding to preset scanned organ categories;

[0016] Extract multiple video frames from each segment, obtain the video attribute parameters of the video frames, and obtain the key video frames in the segment based on the video attribute parameters;

[0017] The keyframe index is established based on the key video frames and the timeline index of the medical image video data to be edited.

[0018] Further, the step of extracting multiple video frames from each segment, obtaining video attribute parameters of the video frames, and obtaining key video frames in the segment based on the video attribute parameters specifically includes:

[0019] After performing bone stripping and linear registration on the video frames, the pixel value of the preset position pixel in the first video frame of the segment is calculated.

[0020] The score of the first video frame is obtained based on the pixel values ​​of the preset position pixels in the first video frame and the pixel values ​​of the preset position pixels in the standard image.

[0021] Use the first video frame as the current video frame;

[0022] Calculate the image similarity value between the next video frame and the current video frame in the segment, compare the image similarity value with a preset similarity threshold, and then take the first video frame as the current video frame, until every video frame extracted from the segment has been traversed.

[0023] When the image similarity value is greater than the preset similarity threshold, the score of the next video frame is obtained based on the pixel value of the preset position pixel in the next video frame of the current video frame and the pixel value of the preset position pixel in the standard image.

[0024] When the score of the next video frame after the current video frame is higher than that of the current video frame, the next video frame after the current video frame is taken as the key video frame.

[0025] Otherwise, the current video frame is used as the key video frame;

[0026] When the image similarity value is less than the preset similarity threshold, the score of the current video frame is obtained based on the pixel value of the preset position pixel of the next video frame of the current video frame and the pixel value of the preset position pixel in the standard image, and the current video frame is used as the second key video frame in the segment.

[0027] Furthermore, establishing the keyframe index based on the key video frames and the timeline index of the medical image video data to be edited specifically includes:

[0028] The key video frames are displayed in chronological order on the timeline index of the medical image video data to be edited, thus obtaining the key frame index.

[0029] Furthermore, the step of editing the medical image video data to be edited based on the keyframe index specifically includes:

[0030] The start and end positions of the video clips are determined based on the keyframe indexes.

[0031] The video editing of the medical image video data to be edited is completed according to the start and end positions of the video clip, and the keyframe index is inserted as a tag into the completed medical image video data.

[0032] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0033] A smart editing application device that uses image recognition includes,

[0034] The acquisition module is used to acquire medical image video data to be edited;

[0035] The recognition module is used to perform grouped image recognition on the medical image video data to be edited to obtain keyframe indexes;

[0036] The editing module is used to edit the medical image video data to be edited based on the keyframe index.

[0037] Furthermore, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an intelligent editing application method based on image recognition as described in any of the above technical solutions.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an intelligent editing application method based on image recognition as described in any of the above technical solutions.

[0039] The advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating an intelligent editing application method based on image recognition according to the present invention.

[0042] Figure 2 This is a schematic diagram of a module of an intelligent editing application device based on image recognition according to the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] like Figure 1 The intelligent editing application method based on image recognition described in this embodiment of the invention includes the following steps:

[0045] 110. Obtain the medical image video data to be edited.

[0046] It should be understood that medical imaging video data includes images and videos obtained from various examinations such as CT and PET scans, as well as images and videos taken during surgical procedures.

[0047] 120. Perform grouped image recognition on the medical image video data to be edited to obtain keyframe indexes.

[0048] It should be understood that there are many ways to group medical imaging video data, such as by time or by the tissue site being examined.

[0049] 130. Edit the medical image video data to be edited based on the keyframe index.

[0050] It should be understood that video editing methods involve the field of artificial intelligence technology, using AI to edit videos. In this embodiment, it can be achieved through computer vision technology within artificial intelligence. Computer vision (CV) is a science that studies how machines "see," specifically, it refers to machine vision that uses cameras and computers to replace human eyes for target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0051] In this embodiment, due to the special nature of medical imaging videos, the video editing scheme only considers visual information, and the accuracy of image recognition further affects the editing effect of medical imaging videos.

[0052] Furthermore, step 120 specifically includes:

[0053] Random noise in the medical image video data is eliminated using the k-nearest neighbor average method.

[0054] Image enhancement processing of medical video data is performed using histogram equalization and logarithmic transformation.

[0055] The medical image video data is segmented into segments corresponding to preset scanned organ categories.

[0056] Extract multiple video frames from each segment, obtain the video attribute parameters of the video frames, and obtain the key video frames in the segment based on the video attribute parameters.

[0057] The keyframe index is established based on the key video frames and the timeline index of the medical image video data to be edited.

[0058] Further, step 120, which involves extracting multiple video frames from each segment, obtaining video attribute parameters of the video frames, and obtaining key video frames in the segment based on the video attribute parameters, specifically includes:

[0059] After performing bone stripping and linear registration on the video frames, the pixel value of the preset position pixel in the first video frame of the segment is calculated.

[0060] The score of the first video frame is obtained based on the pixel values ​​of the preset position pixels in the first video frame and the pixel values ​​of the preset position pixels in the standard image.

[0061] Use the first video frame as the current video frame;

[0062] Calculate the image similarity value between the next video frame and the current video frame in the segment, compare the image similarity value with a preset similarity threshold, and then take the first video frame as the current video frame, until every video frame extracted from the segment has been traversed.

[0063] When the image similarity value is greater than the preset similarity threshold, the score of the next video frame is obtained based on the pixel value of the preset position pixel in the next video frame of the current video frame and the pixel value of the preset position pixel in the standard image.

[0064] When the score of the next video frame after the current video frame is higher than that of the current video frame, the next video frame after the current video frame is taken as the key video frame.

[0065] Otherwise, the current video frame is used as the key video frame;

[0066] When the image similarity value is less than the preset similarity threshold, the score of the current video frame is obtained based on the pixel value of the preset position pixel of the next video frame of the current video frame and the pixel value of the preset position pixel in the standard image, and the current video frame is used as the second key video frame in the segment.

[0067] It should be understood that the sensory information in this application includes, but is not limited to, information such as the clarity, saturation, color richness, and brightness of video frames in medical imaging videos.

[0068] The score refers to the score obtained after evaluating the medical value of each video frame in a medical imaging video.

[0069] It should be understood that video frames can be scored based on one type of video attribute parameter in the video to be edited, or multiple video attribute parameters can be used to score video frames.

[0070] It should be understood that video frames with higher scores have better medical reference and learning value, therefore, video frames with higher scores are used as keyframes.

[0071] The keyframe index is obtained by inserting the keyframe into the timeline of the corresponding video.

[0072] The key video frames are displayed in chronological order on the timeline index of the medical image video data to be edited, thus obtaining the key frame index.

[0073] Furthermore, step 130 specifically includes,

[0074] The start and end positions of the video clips are determined based on the keyframe indexes.

[0075] The video editing of the medical image video data to be edited is completed according to the start and end positions of the video clip, and the keyframe index is inserted as a tag into the completed medical image video data.

[0076] It should be understood that the embodiments of the present invention edit the video data to be edited according to the index of the video playback timeline. The video playback timeline index generated by the image recognition result assists in video editing, eliminating the need for editors to watch the entire video content or watch the video repeatedly to complete the video editing operation, thus improving the efficiency of video editing.

[0077] The keyframe index also establishes a correspondence with the video frames in the video data, which can directly locate the corresponding video frames for easy editing, saving editors from manual operations. This not only improves the efficiency of video editing but also enhances the operating experience for editors, making it more convenient.

[0078] Furthermore, based on the above embodiments, video frames and their video attribute parameters can also be input into a trained neural network to obtain scores for the video frames, specifically including:

[0079] The video frame is input into a trained neural network for feature extraction to obtain the feature vector of the video frame.

[0080] The video attribute parameters of the video frame are vectorized to obtain the video attribute parameter vector;

[0081] The interpolation norm of the video attribute parameter vector and the feature vector of the video frame is obtained, and then the sigmoid function is applied through a fully connected layer to obtain the score of the video frame.

[0082] It should be understood that convolutional neural networks are a type of feedforward neural network that includes convolutional computation and has a deep structure, and are widely used in various fields such as image recognition, speech recognition, and natural language recognition.

[0083] In convolutional neural network (CNN) applications, neural network processors are the preferred choice due to their faster processing speed, especially in applications with high real-time requirements. 3D convolutional neural networks specifically refer to multi-dimensional (more than two dimensions) neural networks. Compared to traditional two-dimensional neural networks, multi-dimensional neural networks add one or more dimensions to the two-dimensional network. For example, in video processing, a time dimension is introduced, and multiple frames of video data are input for convolution.

[0084] The C3D (Convolutional 3D) model adds a temporal dimension to the spatial domain of traditional two-dimensional convolutional neural networks, enabling better extraction of temporal features of behavior and offering faster speeds compared to methods like the two-stream method. While C3D convolution can consider features in both spatial and temporal dimensions simultaneously, the computational cost and model storage requirements are still too high. Therefore, the P3D (Pseudo-3D ResNet) network modifies the three-dimensional convolution by dividing a t×m×n convolutional kernel into a t×1×1 kernel in the temporal domain and a 1×m×n kernel in the spatial domain. This reduces computational and storage requirements, allowing for the training of three-dimensional neural networks on larger underlying network models.

[0085] P3D convolutional neural network is a widely used 3D convolutional neural network model. The convolutional layers in a P3D convolutional neural network include two types of convolutional kernels: the first kernel of the first type supports both width and height dimensions, while the second kernel of the second type supports the time dimension.

[0086] This embodiment can employ any of the convolutional neural networks described above, or convolutional neural networks not described herein; this application makes no limitation in this regard. Furthermore, the training methods for convolutional neural networks are currently very mature.

[0087] The intelligent video editing application method based on image recognition proposed in the above embodiments includes: acquiring medical image video data to be edited; performing grouped image recognition on the medical image video data to be edited to obtain keyframe indexes; and editing the medical image video data to be edited based on the keyframe indexes. This invention eliminates the need for editors with medical knowledge to perform editing, and eliminates the need for editors to watch the entire video or watch it repeatedly to complete the video editing operation, thus improving the efficiency of video editing.

[0088] like Figure 2 As shown, a smart editing application device based on image recognition includes,

[0089] The acquisition module is used to acquire medical image video data to be edited;

[0090] The recognition module is used to perform grouped image recognition on the medical image video data to be edited to obtain keyframe indexes;

[0091] The editing module is used to edit the medical image video data to be edited based on the keyframe index.

[0092] Furthermore, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of an intelligent editing application method based on image recognition as described in any of the above technical solutions.

[0093] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an intelligent editing application method based on image recognition as described in any of the above technical solutions.

[0094] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0100] Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0102] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent video editing applications based on image recognition, characterized in that, This includes acquiring medical image video data to be edited; The medical image video data to be edited is grouped and image recognition is performed to obtain the keyframe index; Editing the medical image video data to be edited based on the keyframe index, specifically including: performing grouped image recognition on the medical image video data to be edited to obtain the keyframe index, specifically including: eliminating random noise in the medical image video data using the k-nearest neighbor average method; performing image enhancement processing on the medical image video data through histogram equalization and logarithmic transformation; segmenting the medical image video data into segments corresponding to preset scanned organ categories; extracting multiple video frames from each segment, obtaining video attribute parameters of the video frames, and obtaining key video frames in the segment based on the video attribute parameters; establishing the keyframe index based on the key video frames and the time axis index of the medical image video data to be edited, specifically including: after performing bone stripping and linear registration processing on the video frames, calculating the pixel value of a preset position pixel in the first video frame of the segment; and calculating the pixel value of the preset position pixel in the first video frame and the preset position pixel in the standard image. The pixel values ​​are used to obtain the score of the first video frame; the first video frame is taken as the current video frame; the image similarity value between the next video frame in the segment and the current video frame is calculated, and the image similarity value is compared with a preset similarity threshold. Then, the first video frame is taken as the current video frame, and this process is repeated until every video frame extracted from the segment has been traversed; when the image similarity value is greater than the preset similarity threshold, the pixel values ​​of the preset position pixels in the next video frame of the current video frame and the pixel values ​​of the preset position pixels in the standard image are used to determine the score. The score of the next video frame after the current video frame is obtained; when the score of the next video frame after the current video frame is higher than that of the current video frame, the next video frame after the current video frame is taken as the key video frame; otherwise, the current video frame is taken as the key video frame; when the image similarity value is less than the preset similarity threshold, the score of the current video frame is obtained according to the pixel value of the preset position pixel of the next video frame after the current video frame and the pixel value of the preset position pixel in the standard image, and the current video frame is taken as the second key video frame in the segment.

2. The intelligent editing application method based on image recognition according to claim 1, characterized in that, The process of establishing the key frame index based on the key video frames and the timeline index of the medical image video data to be edited specifically includes: displaying the key video frames in chronological order on the timeline index of the medical image video data to be edited to obtain the key frame index.

3. The intelligent editing application method based on image recognition according to claim 1, characterized in that, The step of editing the medical image video data to be edited based on the keyframe index specifically includes: determining the start and end positions of the video editing according to the keyframe index; completing the video editing of the medical image video data to be edited according to the start and end positions of the video editing; and inserting the keyframe index as a tag into the edited medical image video data.

4. A smart editing application device based on image recognition, characterized in that, Includes an acquisition module, used to acquire medical image video data to be edited; The recognition module is used to perform grouped image recognition on the medical image video data to be edited to obtain keyframe indexes; The editing module is used to edit the medical image video data to be edited based on the keyframe index; the recognition module is specifically used to: eliminate random noise in the medical image video data using the k-nearest neighbor average method; perform image enhancement processing on the medical image video data through histogram equalization and logarithmic transformation; segment the medical image video data into segments corresponding to preset scanned organ categories; extract multiple video frames from each segment, obtain video attribute parameters of the video frames, and obtain key video frames in the segment based on the video attribute parameters; establish the keyframe index based on the key video frames and the time axis index of the medical image video data to be edited; after performing bone stripping and linear registration processing on the video frames, calculate the pixel value of the preset position pixel in the first video frame of the segment; obtain the score of the first video frame based on the pixel value of the preset position pixel in the first video frame and the pixel value of the preset position pixel in the standard image; take the first video frame as the current video frame; calculate the image phase of the next video frame of the current video frame in the segment and the image phase of the current video frame. The image similarity value is calculated and compared with a preset similarity threshold. The first video frame is then used as the current video frame, and this process is repeated until all extracted video frames in the segment have been traversed. When the image similarity value is greater than the preset similarity threshold, the score of the next video frame is obtained based on the pixel values ​​of preset position pixels in the next video frame of the current video frame and the pixel values ​​of preset position pixels in the standard image. When the score of the next video frame of the current video frame is higher than that of the current video frame, the next video frame of the current video frame is used as the key video frame. Otherwise, the current video frame is used as the key video frame. When the image similarity value is less than the preset similarity threshold, the score of the current video frame is obtained based on the pixel values ​​of preset position pixels in the next video frame of the current video frame and the pixel values ​​of preset position pixels in the standard image, and the current video frame is used as the second key video frame in the segment.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the intelligent editing application method based on screen recognition as described in any one of claims 1-3.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of an intelligent editing application method based on screen recognition as described in any one of claims 1-3.

Images