A method and system for image and video retrieval

Abstract

The application relates to the field of network security monitoring, and discloses a method and system for picture and video retrieval, which are characterized by the following steps: setting a frame image acquisition module, a picture tracing module, an implantation identification module and an object processing module, performing continuity judgment on image frames of video content, effectively responding to and extracting the content randomly implanted in the video, performing network retrieval on the features of the extracted safety identification image, effectively acquiring the implanted content in the video according to the image features of the source image, and judging the network security of the related implanted content. Compared with the traditional method, the method is more efficient and reliable, and has higher implanted content identification effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network security monitoring, and specifically to a method and system for picture and video retrieval. BACKGROUND

[0002] The rapid development of network technology has led to the current information explosion era. Through intelligent devices connected to the Internet, a large amount of information content can be obtained and browsed every day. However, a large amount of information content inevitably contains a large amount of information with undesirable guidance, which has a certain impact on society and life.

[0003] In the prior art, the identification of implanted advertisements in video and image content in the network is mostly based on the random selection of several frame images from the video for content identification and judgment. However, this method has the problem that the guidance content implanted in the video for a short time is easily missed, leading to the inability to judge it, and thus leaving a security risk. Moreover, the direct identification method often has the problem of inaccurate content identification or the inability to effectively judge whether the content is implanted content, resulting in poor use efficiency. SUMMARY

[0004] The present application aims to provide a method and system for picture and video retrieval to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] A system for picture and video retrieval, comprising:

[0007] A frame image acquisition module for judging the continuity of image frames based on the time axis for video content. If the continuity judgment result is that the image frames are not continuous, the image frames located behind the time axis are taken as safety identification images. The continuity judgment is used to judge the content matching of adjacent frames.

[0008] A picture tracing module for feature extraction processing of the safety identification images to obtain corresponding image retrieval features. The network image library is retrieved according to the image retrieval features to obtain source images that achieve a preset coincidence rate with the image retrieval features.

[0009] An implanted identification module for analyzing and extracting the distinguishing features of the safety identification images according to the source images, generating implanted distinguishing features of the safety identification images, identifying the content of the implanted distinguishing features, and generating a content judgment result.

[0010] An object processing module is configured to perform content security rating on the content judgment result based on a preset security rating standard, and generate a corresponding security warning feedback, which is used to be bound with the security identification image in a link, and the security warning feedback is output when the security identification image is output.

[0011] As a further scheme of the present application, the implant identification module further comprises a security screening unit.

[0012] The security screening unit is configured to judge the coincidence rate, and if the coincidence rate reaches a preset error allowable coincidence rate, the step of performing analysis and extraction of the distinguished features of the security identification image is no longer performed, and the error allowable coincidence rate is used to represent that the image retrieval features of the security identification image are consistent with those of the source image.

[0013] As a further scheme of the present application, the frame image acquisition module specifically comprises:

[0014] A feature selection unit is configured to acquire frames of the video content based on a time axis, divide regions of the frames based on image features, and acquire a plurality of feature images, and the plurality of feature images correspond to different plane motion states on the time axis.

[0015] A special diagnosis determination unit is configured to judge the continuity of the plurality of feature images, and if there is no adjacent frame of the image frame at a previous moment on the time axis, the image frame of the feature image is a security identification image.

[0016] As a further scheme of the present application, the implant identification module comprises:

[0017] An implant content separation unit is configured to acquire the source image and the security identification image, judge the coincidence of the image retrieval features of the security identification image based on the image retrieval features of the source image, remove the coincident part of the image retrieval features of the security identification image and the source image, and acquire implant distinguished features distinguished from the source image.

[0018] An implant content judgment unit is configured to perform content recognition on the implant distinguished features by using a preset content recognition program, and acquire a plurality of recognition content labels, and the content judgment result comprises the recognition content labels.

[0019] As a further scheme of the present application, the picture tracing module is provided with a tracing determination unit.

[0020] The traceability determination unit is configured to arrange the source images in descending order according to the coincidence rate and the publishing time information of the source images when a plurality of the image retrieval features reach a preset coincidence rate, and select the source image with the highest coincidence rate and the longest publishing time from the current time.

[0021] The embodiment of the present application aims to provide a picture and video retrieval-based method, comprising the steps of:

[0022] The continuity of the image frames of the video content is determined based on the time axis, and if the continuity determination result is that the image frames are not continuous, the image frame located at the rear of the time axis is taken as a safety identification image, and the continuity determination is used to determine the content matching of adjacent frames.

[0023] The safety identification image is subjected to feature extraction processing to obtain corresponding image retrieval features, the network image library is retrieved according to the image retrieval features, and source images reaching a preset coincidence rate with the image retrieval features are obtained.

[0024] The safety identification image is subjected to analysis and extraction of distinguishing features based on the source image, implantation distinguishing features of the safety identification image are generated, content recognition determination is performed on the implantation distinguishing features, and a content determination result is generated.

[0025] The content determination result is subjected to content safety rating based on a preset safety rating standard, and corresponding safety warning feedback is generated, the safety warning feedback is used to establish link binding with the safety identification image, the safety warning feedback is output when the safety identification image is output, and the link binding is also used to receive safety rating feedback of the safety warning feedback by a user.

[0026] As a further scheme of the present application, the step of analyzing and extracting distinguishing features of the safety identification image based on the source image further comprises a pre-determination step:

[0027] The coincidence rate is determined, and if the coincidence rate reaches a preset error allowable coincidence rate, the step of analyzing and extracting distinguishing features of the safety identification image is no longer performed, and the error allowable coincidence rate is used to represent that the image retrieval features of the safety identification image and the source image are consistent.

[0028] As a further scheme of the present application, the step of determining the continuity of the image frames of the video content based on the time axis, if the continuity determination result is that the image frames are not continuous, the image frame located at the rear of the time axis is taken as a safety identification image specifically comprises:

[0029] Frame acquisition is performed on the video content based on a time axis, and region division of image features is performed on the frames to obtain a plurality of feature images, wherein the plurality of feature images correspond to different planar motion states on the time axis respectively.

[0030] Continuity judgment is performed on the plurality of feature images, and if the image frame does not exist on the adjacent frame at the previous moment on the time axis, the image frame where the feature image is located is a security authentication image.

[0031] As a further scheme of the present application, the step of generating the implanted distinguishing feature of the security authentication image by performing analysis and extraction on the distinguishing feature of the security authentication image according to the source image, performing content recognition judgment on the implanted distinguishing feature, and generating a content judgment result specifically includes:

[0032] The source image and the security authentication image are obtained, coincidence judgment is performed on the image retrieval feature of the security authentication image according to the image retrieval feature of the source image, and the overlapping part of the image retrieval feature of the security authentication image and the source image is removed to obtain an implanted distinguishing feature that is different from the source image.

[0033] Content recognition is performed on the implanted distinguishing feature by a preset content recognition program to obtain a plurality of recognition content labels, and the content judgment result includes the recognition content labels.

[0034] As a further scheme of the present application, the step of obtaining the source image with the image retrieval feature reaching a preset coincidence rate further includes:

[0035] When a plurality of source images with the image retrieval feature reaching a preset coincidence rate are retrieved and obtained, the plurality of source images are arranged in descending order according to the coincidence rate and the publishing time information of the source images, and the source image with the highest coincidence rate and the longest publishing time from the current moment is selected.

[0036] Compared with the prior art, the present application has the beneficial effects that the present application can perform continuity judgment on image frames of video content, so that the randomly implanted content in the video can be effectively responded and extracted, and the security authentication image extracted can be subjected to feature network retrieval, so that the implanted content in the video can be effectively obtained according to the image features of the source image, and the network security of the related implanted content can be informed by judgment, which is more efficient and reliable than the traditional method and has higher implanted content recognition effect. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a component block diagram of a system based on picture and video retrieval.

[0038] Figure 2A constituent block diagram of a frame image acquisition module in a picture and video retrieval-based system.

[0039] Figure 3 A constituent block diagram of an implantation identification module in a picture and video retrieval-based system.

[0040] Figure 4 A flow block diagram of a picture and video retrieval-based method. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0042] The specific implementation of the present application will be described in detail below with reference to specific embodiments.

[0043] As Figure 1 described, a picture and video retrieval-based system provided by an embodiment of the present application includes the following steps:

[0044] A frame image acquisition module 100 is configured to judge the continuity of image frames of video content based on a time axis. If the continuity judgment result is that the image frames are discontinuous, the image frame located at the rear of the time axis is taken as a safety identification image. The continuity judgment is used to judge the content matching of adjacent frames.

[0045] A picture tracing module 300 is configured to perform feature extraction processing on the safety identification image to obtain corresponding image retrieval features, perform retrieval on a network image library according to the image retrieval features, and obtain a source image that achieves a preset coincidence rate with the image retrieval features.

[0046] An implantation identification module 500 is configured to perform analysis and extraction of distinguishing features of the safety identification image according to the source image, generate implantation distinguishing features of the safety identification image, perform content recognition judgment on the implantation distinguishing features, and generate a content judgment result.

[0047] An object processing module 700 is configured to perform content safety rating on the content judgment result based on a preset safety rating standard, generate a corresponding safety warning feedback, and establish link binding with the safety identification image. When the safety identification image is output, the safety warning feedback is output. The link binding is also used to receive safety rating feedback of the safety warning feedback by a user.

[0048] In this embodiment, a system based on picture and video retrieval is given. Through the setting of relevant modules and methods, the continuity of image frames of video content can be judged, so that the randomly implanted content in the video can be effectively responded and extracted, and through the network retrieval of the extracted security identification image, the source image can be obtained, so that according to the difference of the image features of the source image, the implanted content can be effectively obtained, so that the implanted content can be judged for content security and a safety warning is generated for informing the browsing personnel. Specifically, the frame image acquisition module 100 can identify when a certain frame of the video is implanted with other content (including the change image of the video content itself, which will also be identified), the picture tracing module 300 extracts the features of the identified content and traces the network to obtain the source image, so that the implantation identification module 500 can further judge whether the content is implanted content (or whether the content itself can be safely spread, etc.), and the safety of the implanted content, so that the object processing module 700 is marked and warned, for example, when the user watches the video to this implanted content or watches the picture (when the frame image acquisition module 100 does not need to work when the image is), the relevant content safety warning is popped up, which can effectively avoid the negative impact of network fraud and bad guidance of network content on the browsing personnel.

[0049] As another preferred embodiment of the present application, the implantation identification module 500 further comprises a safety screening unit.

[0050] The safety screening unit is configured to judge the coincidence rate. If the coincidence rate reaches a preset error allowable coincidence rate, the step of analyzing and extracting the distinguished features of the security identification image is no longer performed. The error allowable coincidence rate is used to represent that the image retrieval features of the security identification image and the source image are consistent.

[0051] In this embodiment, the error allowable coincidence rate is less than 100%, because it includes the error of feature recognition in data processing, etc. When two contents are completely consistent, it can be judged that they are not implanted with relevant advertisements, guidance, etc. However, the content itself may also be an advertisement itself, so the user can also set a safety mark for the network content. Therefore, after the source image is retrieved, the safety mark of the source image can be obtained, and the safety warning feedback can also be generated according to the safety mark of the source image.

[0052] As shown in Figure 2 As another preferred embodiment of the present application, the frame image acquisition module 100 specifically comprises:

[0053] The feature selection unit 101 is configured to acquire frames of the video content based on a time axis, and divide the frames into regions of image features to obtain a plurality of feature images, wherein the plurality of feature images correspond to different planar motion states on the time axis respectively.

[0054] The special diagnosis determination unit 102 is configured to judge the continuity of the plurality of feature images, and if there is no adjacent frame of the image frame at a previous moment on the time axis, the image frame of the feature image is a safety identification image.

[0055] In this embodiment, the feature selection unit 101 is supplemented because the video includes a plurality of image elements, and the motion states of the image elements can be quite different, even in the opposite direction (for example, the background image element moves to the left side, and the image element of the person moves rapidly and obliquely, etc.). If only the frames are judged, the overall continuity of the frame images can be discontinuous, and therefore different elements need to be divided and processed.

[0056] As shown in Figure 3 As another preferred embodiment of the present application, the implant identification module 500 includes:

[0057] The implant content separation unit 501 is configured to acquire the source image and the safety identification image, judge the coincidence of the image retrieval features of the safety identification image based on the image retrieval features of the source image, remove the overlapping part of the image retrieval features of the safety identification image and the source image, and obtain implant distinguishing features different from the source image.

[0058] The implant content judgment unit 502 is configured to identify the implant distinguishing features by a preset content recognition program to obtain a plurality of recognition content labels, and the content judgment result includes the recognition content labels.

[0059] In this embodiment, the implant identification module 500 is functionally divided, wherein the implant content separation unit 501 can be simply understood as overlapping the source image after finding the source image, and the part that does not overlap is the subsequent implanted content. Therefore, the implant content judgment unit 502 obtains a plurality of recognition content labels by searching or recognizing the content (including image recognition, person recognition, text recognition, etc.), and the content label corresponds to the content safety rating, that is, whether the content will have adverse effects in the spread, etc.

[0060] As another preferred embodiment of the present application, the picture tracing module 300 is provided with a tracing determination unit.

[0061] The traceability determination unit is configured to arrange the source images in descending order according to the coincidence rate and the publishing time information of the source images when a plurality of the source images with the image retrieval features reaching the preset coincidence rate are retrieved, and select the source image with the highest coincidence rate and the longest publishing time from the current time.

[0062] In the embodiment, in the retrieval process, there are usually more retrieval objects, some of which may have been implanted with content or modified, and thus the most original source image is needed for subsequent content judgment, which requires a selection process as provided herein.

[0063] As shown in Figure 4 The application further provides a picture and video retrieval-based method, which comprises the following steps:

[0064] S200, performing image frame continuity judgment on the video content based on a time axis, taking the image frame located at the rear of the time axis as a safety identification image if the continuity judgment result is that the image frames are discontinuous, and the continuity judgment is used to judge the content matching of adjacent frames.

[0065] S400, performing feature extraction processing on the safety identification image to obtain corresponding image retrieval features, retrieving a network image library according to the image retrieval features, and obtaining source images with the image retrieval features reaching a preset coincidence rate.

[0066] S600, analyzing and extracting the distinguishing features of the safety identification image based on the source images, generating implanted distinguishing features of the safety identification image, performing content recognition judgment on the implanted distinguishing features, and generating a content judgment result.

[0067] S800, performing content safety rating on the content judgment result based on a preset safety rating standard, generating a corresponding safety warning feedback, the safety warning feedback is used to establish a link binding with the safety identification image, the safety warning feedback is output when the safety identification image is output, and the link binding is also used to receive a safety rating feedback of the safety warning feedback by a user.

[0068] As another preferred embodiment of the application, the step of analyzing and extracting the distinguishing features of the safety identification image based on the source images further comprises a pre-judgment step:

[0069] judging the coincidence rate, if the coincidence rate reaches a preset error allowable coincidence rate, the step of analyzing and extracting the distinguishing features of the safety identification image is no longer performed, and the error allowable coincidence rate is used to represent that the image retrieval features of the safety identification image and the source image are consistent.

[0070] As another preferred embodiment of the present application, the step of judging the continuity of the image frames of the video content based on the time axis, if the result of the continuity judgment is that the image frames are not continuous, taking the image frame located behind the time axis as the security identification image specifically includes:

[0071] Based on the time axis, the frames of the video content are acquired, and the image features of the frames are divided into regions to obtain a plurality of feature images, and the plurality of feature images correspond to different plane motion states on the time axis, respectively.

[0072] The continuity of the plurality of feature images is judged, and if the image frame does not exist on the adjacent frame of the time axis at the previous moment, the image frame of the feature image is a security identification image.

[0073] As another preferred embodiment of the present application, the step of analyzing and extracting the distinguished features of the security identification image according to the source image, generating the implanted distinguished features of the security identification image, judging the content of the implanted distinguished features, and generating the content judgment result specifically includes:

[0074] The source image and the security identification image are acquired, the image retrieval features of the security identification image are judged to coincide with the image retrieval features of the source image, and the coincident part of the image retrieval features of the security identification image and the source image is removed to obtain the implanted distinguished features that are different from the source image.

[0075] The content of the implanted distinguished features is identified by a preset content identification program to obtain a plurality of identification content labels, and the content judgment result includes the identification content labels.

[0076] As another preferred embodiment of the present application, the step of acquiring the source image with the image retrieval features reaching a preset coincidence rate further includes:

[0077] When a plurality of source images with the image retrieval features reaching a preset coincidence rate are retrieved and acquired, the plurality of source images are arranged in descending order according to the coincidence rate and the publishing time information of the source images, and the source image with the highest coincidence rate and the longest publishing time from the current time is selected.

[0078] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0079] Other embodiments of the present disclosure will be apparent to those skilled in the art with the accomplishment of the present disclosure as reflected in the specification and embodiments. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure following the general principles of the present disclosure and including common knowledge or conventional technical means in the art not disclosed in the present disclosure. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

[0080] It should be understood that the present disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A system for image and video retrieval based on, characterized by, The application relates to a video security identification method and device. The method comprises the following steps: a frame image acquisition module is used for judging the continuity of image frames of video content based on a time axis, if the continuity judgment result is that the image frames are discontinuous, the image frame located at the rear of the time axis is taken as a safety identification image, and the continuity judgment is used for judging the content matching of adjacent frames; a picture tracing module is used for carrying out feature extraction processing on the safety identification image to obtain corresponding image retrieval features, searching a network image library according to the image retrieval features, and obtaining source images which reach a preset coincidence rate with the image retrieval features; an implantation identification module is used for analyzing and extracting the distinguishing features of the safety identification image according to the source images, generating implantation distinguishing features of the safety identification image, judging the content of the implantation distinguishing features, and generating a content judgment result; an object processing module is used for judging the content safety rating of the content judgment result based on a preset safety evaluation standard, generating corresponding safety warning feedback, and establishing link binding with the safety identification image, wherein the safety warning feedback is output when the safety identification image is output, and the link binding is also used for receiving the safety rating feedback of the safety warning feedback by a user; the frame image acquisition module specifically comprises: a feature selection unit is used for acquiring frames of the video content based on a time axis, dividing the image features of the frames into regions, obtaining a plurality of feature images, and making the plurality of feature images correspond to different plane motion states on the time axis; a special diagnosis determination unit is used for judging the continuity of the plurality of feature images, if the image frame does not exist on the adjacent frame of the time axis at the previous moment, the image frame of the feature image is a safety identification image; the implantation identification module comprises: an implantation content separation unit is used for obtaining the source images and the safety identification image, judging the coincidence of the image retrieval features of the safety identification image according to the image retrieval features of the source images, removing the coincident part of the image retrieval features of the safety identification image and the source images, and obtaining implantation distinguishing features which are different from the source images; 2. The system for image and video retrieval based on claim 1, characterized in that, an implantation content judgment unit is used for judging the content of the implantation distinguishing features through a preset content recognition program, obtaining a plurality of recognition content labels, and the content judgment result comprises the recognition content labels. The implantation identification module further comprises a safety screening unit; 3. The system for image and video retrieval based on claim 1, characterized in that, the safety screening unit is used for judging the coincidence rate, if the coincidence rate reaches a preset error allowable coincidence rate, the step of analyzing and extracting the distinguishing features of the safety identification image is not executed any more, and the error allowable coincidence rate is used for representing that the image retrieval features of the safety identification image are consistent with the source images. The picture tracing module is provided with a tracing determination unit; the tracing determination unit is used for arranging a plurality of source images in descending order according to the coincidence rate and the publishing time information of the source images when a plurality of source images which reach a preset coincidence rate with the image retrieval features are searched, and selecting the source image with the highest coincidence rate and the longest publishing time from the current moment.

4. A method for image and video retrieval, the method comprising: The method comprises the steps of: Based on the time axis, the continuity of the image frames of the video content is judged. If the continuity judgment result is that the image frames are not continuous, the image frame located behind the time axis is taken as a safety identification image. The continuity judgment is used to judge the content matching of adjacent frames; The feature extraction processing is performed on the safety identification image to obtain the corresponding image retrieval feature. The network image library is retrieved according to the image retrieval feature to obtain a source image that reaches a preset coincidence rate with the image retrieval feature; According to the source image, the implanted distinguishing features of the safety identification image are analyzed and extracted to generate the implanted distinguishing features of the safety identification image. The content of the implanted distinguishing features is identified, and a content judgment result is generated; Based on the preset safety evaluation standard, the content safety rating of the content judgment result is performed to generate a corresponding safety warning feedback. The safety warning feedback is used to establish a link binding with the safety identification image. When the safety identification image is output, the safety warning feedback is output. The link binding is also used to receive the safety rating feedback of the user on the safety warning feedback; The step of judging the continuity of the image frames of the video content based on the time axis, if the continuity judgment result is that the image frames are not continuous, the image frame located behind the time axis is taken as a safety identification image, specifically comprises: Based on the time axis, the frames of the video content are obtained, and the image features of the frames are regionally divided to obtain a plurality of feature images. The plurality of feature images correspond to different plane motion states on the time axis respectively; The continuity of a plurality of the feature images is judged. If the image frame does not exist on the adjacent frame of the time axis at the previous moment, the image frame of the feature image is a safety identification image; The step of analyzing and extracting the distinguishing features of the safety identification image according to the source image to generate the implanted distinguishing features of the safety identification image, identifying the content of the implanted distinguishing features, and generating a content judgment result, specifically comprises: The source image and the safety identification image are obtained. The image retrieval features of the safety identification image are judged to coincide with the image retrieval features of the source image, and the coincident part of the image retrieval features of the safety identification image and the source image is removed to obtain the implanted distinguishing features that are different from the source image; Through a preset content identification program, the content of the implanted distinguishing features is identified to obtain a plurality of identification content tags. The content judgment result includes the identification content tags.

5. The method of retrieving pictures and videos based on the method of claim 4, wherein, The step of analyzing and extracting the distinguishing features of the safety identification image according to the source image further comprises a pre-judgment step: If the coincidence rate reaches a preset error allowable coincidence rate, the step of analyzing and extracting the distinguishing features of the safety identification image is not executed. The error allowable coincidence rate is used to represent that the image retrieval features of the safety identification image and the source image are consistent.

6. The method of retrieving pictures and videos based on the method of claim 4, wherein, The step of obtaining a source image that reaches a preset coincidence rate with the image retrieval feature when executed further comprises: When the source images whose image retrieval features reach the preset coincidence rate are retrieved, the source images are arranged in descending order according to the coincidence rate and the publishing time information of the source images, and the source image with the highest coincidence rate and the longest publishing time from the current time is selected.

Images