Information detection method and device, storage medium, electronic equipment and program product

Abstract

The invention discloses an information detection method and device, a storage medium, electronic equipment and a program product. The method relates to the field of artificial intelligence, and comprises the following steps: in response to an information detection request, performing information extraction on a target page to obtain N pictures and a target text, the information detection request being used for requesting to detect whether public opinion information related to a financial institution exists in the target page; performing feature extraction on the target text to obtain text features; the N pictures are input into a visual feature extractor, target picture features are output, the visual feature extractor comprises M residual networks and T first full connection layers, and each residual network is used for performing feature extraction on one picture; performing feature fusion on the text features and the target picture features to obtain target fusion features; and determining an information detection result based on the target fusion feature. Through the financial public opinion detection method and device, the problem that the detection result is inaccurate when financial public opinion detection is carried out by using single-mode data in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an information detection method, device, storage medium, electronic device, and program product. Background Art

[0002] At present, with the rapid development of machine learning and deep learning technologies, machine learning and deep learning technologies have been put into financial public opinion detection. Most of the existing algorithms only use single-modal data (text) for financial public opinion detection.

[0003] In the related art, for public opinion detection of financial institutions, generally only one picture or one paragraph of text is used as the input of the algorithm for public opinion information detection, and the amount of information in a picture or a paragraph of text is relatively small. Therefore, in the related art, the results of public opinion information detection are inaccurate.

[0004] Currently, no effective solution has been proposed to the problem of inaccurate detection results when using single-modal data for financial public opinion detection in related technologies. Summary of the invention

[0005] The main purpose of this application is to provide an information detection method, device, storage medium, electronic device and program product to solve the problem of inaccurate detection results when using single-modal data for financial public opinion detection in related technologies.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for information detection is provided. The method includes: responding to an information detection request, extracting information from a target page, and obtaining N pictures and a target text, wherein the information detection request is used to request detection of whether the target page has public opinion information related to a financial institution, and N is a positive integer; extracting features from the target text to obtain text features; inputting the N pictures into a visual feature extractor, and outputting target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each of the residual networks is used to extract features from one of the pictures, and the M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain the target picture features, and M and T are positive integers; performing feature fusion on the text features and the target picture features to obtain the target fusion features; based on the target fusion features, determining the information detection result, wherein the information detection result is used to indicate whether the target page has public opinion information related to a financial institution.

[0007] Furthermore, N of the images are input into a visual feature extractor, and target image features are output, including: sorting the N images according to a preset sorting rule to obtain sorted images, wherein the preset sorting rule includes: time order; and inputting the sorted images into the visual feature extractor in sequence, and outputting the target image features.

[0008] Furthermore, the sorted images are sequentially input into the visual feature extractor, and the target image features are output, including: when N is less than M, the sorted images are supplemented in order from the first image until the number of sorted images reaches M, thereby obtaining M sorted images; the sorted M images are sequentially input into the visual feature extractor, and the target image features are output.

[0009] Further, the M sorted pictures are sequentially input into the visual feature extractor to output the target picture feature, including: among the M sorted pictures, the first picture is input into the first residual network to output the first picture feature, the second picture is input into the second residual network to output the second picture feature, and the first first fully connected layer is used to perform feature fusion on the first picture feature and the second picture feature to obtain a first fused feature, wherein the dimensions of the first picture feature, the second picture feature and the first fused feature are the same; among the M sorted pictures, the third picture is input into the third residual network to output the third picture feature, and the second first fully connected layer is used to perform feature fusion on the third picture feature and the first fused feature to obtain a second fused feature, wherein the dimensions of the third picture feature, the first fused feature and the second fused feature are the same; based on the second fused feature, the fourth to the last residual network, the fourth to the last picture in the sorted M pictures, and the third to the last first fully connected layer, the target picture feature is determined, and the target picture feature is output through the last first fully connected layer.

[0010] Furthermore, information is extracted from the target page to obtain N pictures, including: when the target page includes a video, the total playback time of the video is determined, and based on the ratio of the total playback time to N, the target time interval is determined; the video is intercepted according to the target time interval to obtain N pictures.

[0011] Furthermore, the dimensions of the text features, the target image features and the target fusion features are the same, and feature fusion is performed on the text features and the target image features to obtain the target fusion features, including: inputting the text features and the target image features into a second fully connected layer; and outputting the target fusion features.

[0012] Furthermore, based on the target fusion feature, an information detection result is determined, including: inputting the target fusion feature into a detector, and using a normalized exponential function in the detector to determine the information detection result, wherein the detector is used to detect whether there is public opinion information related to the financial institution in the target fusion feature.

[0013] In order to achieve the above-mentioned purpose, according to another aspect of the present application, an information detection device is provided. The device includes: a first extraction unit, which is used to respond to an information detection request, extract information from a target page, and obtain N pictures and a target text, wherein the information detection request is used to request to detect whether the target page has public opinion information related to a financial institution, and N is a positive integer; a second extraction unit, which is used to extract features from the target text to obtain text features; a processing unit, which is used to input the N pictures into a visual feature extractor and output target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each of which is used to extract features from one of the pictures, and M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain the target picture features, and M and T are positive integers; a fusion unit, which is used to perform feature fusion on the text features and the target picture features to obtain target fusion features; a determination unit, which is used to determine the information detection result based on the target fusion features, wherein the information detection result is used to indicate whether the target page has public opinion information related to a financial institution.

[0014] Furthermore, the processing unit includes: a sorting subunit, used to sort the N pictures according to preset sorting rules to obtain sorted pictures, wherein the preset sorting rules include: time order; a first processing subunit, used to input the sorted pictures into the visual feature extractor in sequence and output the target picture features.

[0015] Furthermore, the first processing sub-unit includes: a first processing module, which is used to, when N is less than M, sequentially supplement the sorted pictures starting from the first picture until the number of sorted pictures reaches M, thereby obtaining M sorted pictures; and a second processing module, which is used to input the sorted M pictures into the visual feature extractor in sequence and output the target picture features.

[0016] Further, the second processing module includes: a first processing submodule, which is used to input the first picture into the first residual network among the sorted M pictures, output the first picture feature, input the second picture into the second residual network, output the second picture feature, and use the first first fully connected layer to perform feature fusion on the first picture feature and the second picture feature to obtain a first fusion feature, wherein the dimensions of the first picture feature, the second picture feature and the first fusion feature are all the same; a second processing submodule, which is used to input the third picture into the third residual network among the sorted M pictures, output the third picture feature, and use the second first fully connected layer to perform feature fusion on the third picture feature and the first fusion feature to obtain a second fusion feature, wherein the dimensions of the third picture feature, the first fusion feature and the second fusion feature are the same; a third processing submodule, which is used to determine the target picture feature based on the second fusion feature, the fourth to last residual network, the fourth to last picture among the sorted M pictures, and the third to last first fully connected layer, and output the target picture feature through the last first fully connected layer.

[0017] Furthermore, the first extraction unit includes: a second processing sub-unit, used to determine the total playback time of the video when the target page includes the video, and determine the target time interval based on the ratio of the total playback time to N; an interception sub-unit, used to intercept the video according to the target time interval to obtain N pictures.

[0018] Furthermore, the dimensions of the text features, the target image features and the target fusion features are the same, and the fusion unit includes: a third processing subunit, used to input the text features and the target image features into a second fully connected layer; and output the target fusion features.

[0019] Furthermore, the determination unit includes: a third processing sub-unit, used to input the target fusion feature into the detector, and use the normalized exponential function in the detector to determine the information detection result, wherein the detector is used to detect whether there is public opinion information related to the financial institution in the target fusion feature.

[0020] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the information detection method.

[0021] According to another aspect of the present application, an electronic device is provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the information detection method is executed when the program is running.

[0022] According to another aspect of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions implement the steps of the information detection method when executed by a processor.

[0023] In the present application, in response to an information detection request, information extraction is performed on a target page to obtain N images and a target text, wherein the information detection request is used to request detection of whether the target page has public opinion information related to a financial institution, and N is a positive integer; feature extraction is performed on the target text to obtain text features; the N images are input into a visual feature extractor, and target image features are output, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from an image, and the M first fully connected layers are used to perform feature fusion on the image features extracted by the M residual networks to obtain target image features, and M and T are positive integers; feature fusion is performed on text features and target image features to obtain target fusion features; based on the target fusion features, an information detection result is determined, wherein the information detection result is used to indicate whether the target page has public opinion information related to a financial institution, thereby solving the technical problem of inaccurate detection results in related technologies using single-modal data for financial public opinion detection.

[0024] In the present invention, feature extraction is performed on the text features of the target text, feature extraction is performed on the image features of N pictures through M residual networks in the visual feature extractor, the text features and the image features of multiple pictures are fused, and financial public opinion information detection is performed based on the fused features, thereby avoiding the situation in the related art of performing financial public opinion information detection based on a single picture or a single text, resulting in inaccurate detection results, thereby achieving the technical effect of improving the accuracy of financial public opinion information results. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0026] Figure 1 A hardware structure block diagram of a computer terminal for implementing an information detection method is shown;

[0027] Figure 2 is a flow chart of an information detection method provided according to an embodiment of the present application;

[0028] Figure 3This is a network architecture diagram of a public opinion information detection model provided according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of a multi-graph residual network structure provided according to an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of an information detection device provided according to an embodiment of the present application;

[0031] Figure 6 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] It should be noted that the collected information (including but not limited to user device information, user personal information and public opinion information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with relevant laws, regulations and standards, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation entrances are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions to provide users with corresponding operation entrances for users to choose to agree or refuse the results of automated decision-making; if the user chooses to refuse, the expert decision-making process will be entered.

[0035] Embodiment 1

[0036] According to an embodiment of the present application, a method embodiment of an information detection method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an information detection method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0038] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the information detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned information detection method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0041] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0042] Under the above operating environment, this application provides Figure 2 The information detection method shown. Figure 2 It is a flow chart of the information detection method according to the first embodiment of the present application.

[0043] Step S201, in response to an information detection request, extract information from a target page to obtain N images and target texts, wherein the information detection request is used to request detection of whether the target page has public opinion information related to financial institutions, and N is a positive integer.

[0044] The above-mentioned information detection is used to request to detect whether there is public opinion information related to the financial institution on the target page. The above-mentioned target page may include but is not limited to: a page of a social software, a page of a financial institution's software.

[0045] In this embodiment, pictures in the target page can be extracted to obtain N pictures, and text in the target page can be extracted to obtain target text. In an optional example, the pictures in the target page can also be pictures obtained by capturing the video in the target page. When capturing the video in the target page, the pictures can be captured at equal time intervals to reduce the similarity of the pictures and make the picture information richer.

[0046] Step S202: extract features from the target text to obtain text features.

[0047] In this embodiment, feature extraction can be performed on the target text by a text feature extractor to obtain text features, wherein the text feature extractor may include: a pre-trained BERT model (a language representation model) for text feature extraction.

[0048] For example, this information uses the BERT model to extract text feature representation. Without loss of generality, the text content can be represented as S. The fine-grained text features extracted by the pre-trained BERT model can be represented as:

[0049] t=BERT(S)

[0050] BERT() represents the BERT model, and t represents the extracted text features. Then, the text features can be made into a preset dimension using a fully connected layer, wherein the preset dimension is the same as the dimension of the target image feature in step S203.

[0051] F T =σ(W t ·t)

[0052] Among them, W t represents the weight matrix in the fully connected layer, and σ(·) represents the fully connected operation.

[0053] Step S203, input N pictures into a visual feature extractor, and output target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from a picture, and the M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain target picture features, and M and T are positive integers.

[0054] The above-mentioned visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from a picture, and the M first fully connected layers are used to fuse the picture features extracted by the M residual networks to obtain the target picture features, thereby avoiding the situation where the amount of feature information extracted from a picture is small.

[0055] In this embodiment, the first two pictures can be connected in series in chronological order, and features of the same dimension as the visual features can be obtained through the first fully connected layer. Next, the third feature (the picture feature of the third picture) is connected in series (or called feature fusion) and passed through the first fully connected layer, and it is iterated until all pictures are input. If the number of picture inputs set by the network is greater than the number of visual feature inputs (that is, when M is greater than N), it is necessary to loop the previous steps, and connect the obtained visual features (that is, picture features) in series with the first visual features (picture features of the first picture) for iteration until the number of picture inputs set by the network is met. Since the picture features of the two pictures can be connected in series through a fully connected layer, and then connected in series with the picture features of the third picture through a fully connected layer, and each picture can be feature extracted through a residual network, therefore, M=T+1 can be obtained.

[0056] In this embodiment, feature extraction and feature fusion are performed through multiple pictures to make the information extraction more complete, so as to improve the accuracy of the subsequent public opinion detection results.

[0057] Step S204, performing feature fusion on the text features and the target image features to obtain target fusion features.

[0058] In this embodiment, the text features and the target image features may be fused through a fully connected layer to obtain a target fused feature, and the feature dimensions of the target fused feature, the text features, and the target image features may be the same.

[0059] Step S205: determining an information detection result based on the target fusion feature, wherein the information detection result is used to indicate whether the target page has public opinion information related to the financial institution.

[0060] In this embodiment, the target fusion feature can be input into the public opinion detector, and then the softmax function (normalized exponential function) is directly used to predict financial public opinion. The formula is:

[0061] z=softmax(σ(F f ·W f ))

[0062] Among them, z represents the information detection result of public opinion information, W f is the weight matrix in the fully connected layer, F f Represents the target fusion feature.

[0063] In this embodiment, through the above steps, feature extraction is performed on the text features of the target text, feature extraction is performed on the image features of N pictures through M residual networks in the visual feature extractor, the text features and the image features of multiple pictures are fused, and financial public opinion information detection is performed based on the fused features, thereby avoiding the situation in the related technology of performing financial public opinion information detection based on a single picture or a single text, resulting in inaccurate detection results, thereby achieving the technical effect of improving the accuracy of the financial public opinion information results, and further solving the technical problem of using single modal data for financial public opinion detection in the related technology and inaccurate detection results.

[0064] An optional public opinion information detection model can be used to execute the information detection method provided in this embodiment. Figure 3 is a network architecture diagram of a public opinion information detection model provided in an embodiment of the present application, such as Figure 3 As shown in the figure, it includes: text feature extractor, visual feature extractor, and public opinion detector. Specifically, the text feature extractor can extract text features from financial public opinion information (text), and the visual features (image features) can be extracted using a multi-graph residual network. Then the text features and visual features (target image features) are connected in series and put into the public opinion detector for detection. Figure 3 As shown, FC can represent the fully connected layer in the public opinion detector. In the text feature extractor, the BERT model can be used to extract text feature representation. The image can be obtained by intercepting the video.

[0065] Optionally, in the information detection method provided in the embodiment of the present application, N pictures are input into a visual feature extractor, and target picture features are output, including: sorting the N pictures according to preset sorting rules to obtain sorted pictures, wherein the preset sorting rules include: time order; inputting the sorted pictures into the visual feature extractor in sequence, and outputting target picture features.

[0066] In this embodiment, N pictures can be sorted in chronological order, and the sorted pictures can be input into the residual network in the visual feature extractor in sequence. When the number of residual networks is greater than the number N of pictures actually input, the sorted pictures can be supplemented and input into subsequent residual networks starting from the first picture, so that each residual network can receive a picture, thereby avoiding the features of the first picture from being forgotten.

[0067] The residual network in the visual feature extractor is used to extract features in chronological order, and the fully connected layer in the visual feature extractor is used to fuse features to obtain the target image features.

[0068] Since the input order of the images is based on the time sequence, and adjacent images generally have a contextual relationship, therefore, connecting two adjacent images in series (feature fusion) can make the semantic information more complete, achieving the technical effect of improving the integrity of image feature extraction.

[0069] Optionally, in the information detection method provided in the embodiment of the present application, the sorted pictures are input into the visual feature extractor in sequence, and the target picture features are output, including: when N is less than M, among the sorted pictures, starting from the first picture, supplementing the sorted pictures in sequence until the number of sorted pictures reaches M, to obtain M sorted pictures; the sorted M pictures are input into the visual feature extractor in sequence, and the target picture features are output.

[0070] For example, when the number of images set in the network is less than the actual number of images, the first image can be added to the residual network input in order. For example, the residual network sets the number of input images to 5 (i.e., M=5), but there are actually only 3 images (i.e., N=3) as input (numbered v1 to v3), then the order of input to the residual network is v1, v2, v3, v1, and v2. This is because as the number of network layers increases, the features of the previous input begin to be forgotten, and the residual method can prevent the gradient from disappearing.

[0071] The image is represented as V = [v1, v2, ..., vn], where vi represents the i-th image. The image information is represented by extracting visual features (image features) through the pre-trained VGG-19 (corresponding to the residual network). The feature extraction formula is:

[0072] F vi =VGG(v i )

[0073] VGG() represents the VGG-19 model, and Fvi represents the visual feature representation obtained after the i-th image passes through VGG().

[0074] In this embodiment, F v1 、F v2 、F v3 、F v4 and F v5 The feature fusion is performed to obtain the target image features. By sequentially supplementing the input from the first image to the subsequent residual network in the visual feature extractor, the feature of the first input image is avoided from being forgotten, thus achieving the technical effect of improving the integrity of image feature extraction.

[0075] Optionally, in the information detection method provided in the embodiment of the present application, the sorted M pictures are sequentially input into the visual feature extractor to output the target picture feature, including: among the sorted M pictures, the first picture is input into the first residual network to output the first picture feature, the second picture is input into the second residual network to output the second picture feature, and the first picture feature and the second picture feature are fused by the first first fully connected layer to obtain a first fused feature, wherein the dimensions of the first picture feature, the second picture feature and the first fused feature are the same; among the sorted M pictures, the third picture is input into the third residual network to output the third picture feature, and the second first fully connected layer is used to fuse the third picture feature and the first fused feature to obtain a second fused feature, wherein the dimensions of the third picture feature, the first fused feature and the second fused feature are the same; based on the second fused feature, the fourth to last residual network, the fourth to last picture in the sorted M pictures, and the third to last first fully connected layer, the target picture feature is determined, and the target picture feature is output through the last first fully connected layer.

[0076] Figure 4 is a schematic diagram of a multi-graph residual network structure provided according to an embodiment of the present application, such as Figure 4 As shown in the figure, taking n pictures as an example, picture 1, picture 2, ..., picture n are represented as V = [v1, v2, ..., vn], where vi represents the i-th picture. The picture information is represented by extracting visual features (i.e., picture features) through the pre-trained VGG-19. The formula for visual feature extraction is:

[0077] F vi =VGG(v i )

[0078] Among them, VGG() represents the VGG-19 model, Fvi represents the visual feature representation obtained after the i-th image passes through VGG(), and the extracted image feature Fvi can be fused through the fully connected layer (FC).

[0079] In this embodiment, in chronological order, the first picture is input into the first residual network, the first picture feature is output, the second picture is input into the second residual network, the second picture feature is output, the picture features (first picture features) of the first two pictures and the second picture features are connected in series, and the features of the same dimension as the visual features are obtained through the first fully connected layer (corresponding to the first fusion feature). Next, the third feature (corresponding to the third picture feature) is connected in series and passed through the first fully connected layer, and it is iterated until all pictures are input. In this way, the residual visual features can be finally obtained.

[0080]

[0081] Among them, F Vgg It is represented as residual visual features, and Fci is represented as the i-th fully connected layer.

[0082] The final residual visual feature is passed through a fully connected layer to obtain a feature with the same size as the text feature. Therefore, the obtainable visual feature (corresponding to the target image feature) is expressed as:

[0083] F V =σ(W vf ·F vgg )

[0084] Among them, W vf is the weight matrix in the fully connected layer, F vgg is the residual visual feature representation, F V is the final visual feature.

[0085] By using M residual networks to extract features and fully connected layers to fuse features, we have achieved a feature extraction model using multiple image data, thereby achieving the technical effect of improving the completeness of the extracted image features.

[0086] Optionally, in the information detection method provided in the embodiment of the present application, information is extracted from the target page to obtain N pictures, including: when the target page includes a video, determining the total playback time of the video, and determining the target time interval based on the ratio of the total playback time to N; intercepting the video according to the target time interval to obtain N pictures.

[0087] For example, in terms of visual feature extraction, this embodiment can support two input modes: pictures and videos. For videos, they can be converted into picture information. Pictures in the video can be intercepted at equal time distances. Specifically, S represents the total length of the video, Si represents the i-th picture intercepted, and it is assumed that a total of m pictures need to be intercepted (m>=i), and the time distance of each picture is S / (m+1). For example, if the video length is 5 seconds, 4 pictures need to be intercepted, then pictures are intercepted at the 1st, 2nd, 3rd and 4th seconds. This is because the beginning (0-1 seconds) and end (4-5 seconds) of the video are generally some useless information such as opening remarks, transition pictures or closing remarks. Therefore, intercepting pictures in a manner of equal time intervals (target time intervals) can reduce the similarity of pictures and make the picture information richer.

[0088] Optionally, in the information detection method provided in the embodiment of the present application, the dimensions of the text features, the target image features and the target fusion features are the same, and the text features and the target image features are fused to obtain the target fusion features, including: inputting the text features and the target image features into the second fully connected layer; and outputting the target fusion features.

[0089] In this embodiment, the final text feature and the visual feature (target image) can be connected in series to obtain the fusion feature representation as follows:

[0090]

[0091] By fusing visual features and text features, the technical effect of improving the completeness of information extraction results of the target page is achieved.

[0092] Optionally, in the information detection method provided in the embodiment of the present application, the information detection result is determined based on the target fusion feature, including: inputting the target fusion feature into the detector, and using the normalized exponential function in the detector to determine the information detection result, wherein the detector is used to detect whether there is public opinion information related to financial institutions in the target fusion feature.

[0093] In this embodiment, the target fusion feature can be input into the public opinion detector (corresponding to the detector), and then the softmax function (normalized exponential function) can be directly used to detect financial public opinion information. The detection formula is:

[0094] z=softmax(σ(F f ·W f ))

[0095] Among them, z represents the information detection result of public opinion information, Wf is the weight matrix in the fully connected layer, and F f Represents the target fusion feature.

[0096] It should be noted that, in this embodiment, during the model training process of the public opinion information detection model, the cross entropy loss can be used to predict the target task.

[0097]

[0098] Among them, y i represents the true situation of the i-th data sample (for example, whether there is financial public opinion information), and I represents the total number of data samples.

[0099] In this embodiment, multimodal financial public opinion detection is performed using multiple image data and texts, and the image features of multiple images are spliced ​​using residual splicing. This can lose invalid information in the images and more accurately capture useful information about financial public opinion, thereby improving the accuracy of public opinion information detection results.

[0100] The information detection method provided in the embodiment of the present application, in this embodiment, performs feature extraction on the text features of the target text, performs feature extraction on the image features of N pictures through M residual networks in the visual feature extractor, fuses the text features and the image features of multiple pictures, and performs financial public opinion information detection based on the fused features, thereby avoiding the situation in the related art where financial public opinion information detection is performed based on a single picture or a single text, resulting in inaccurate detection results, thereby achieving the technical effect of improving the accuracy of the financial public opinion information results, and further solving the technical problem in the related art where single modal data is used for financial public opinion detection, resulting in inaccurate detection results.

[0101] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0102] Embodiment 2

[0103] The embodiment of the present application also provides an information detection device. It should be noted that the information detection device of the embodiment of the present application can be used to execute the information detection method provided in the embodiment of the present application. The information detection device provided in the embodiment of the present application is introduced below.

[0104] According to an embodiment of the present application, a device for implementing the above information detection method is also provided. Figure 5 is a schematic diagram of an information detection device provided according to an embodiment of the present application, such as Figure 5 As shown, the device includes: a first extraction unit 51, a second extraction unit 52, a processing unit 53, a fusion unit 54 and a determination unit 55.

[0105] The first extraction unit 51 is used to respond to the information detection request, extract information from the target page, and obtain N pictures and target texts, wherein the information detection request is used to request to detect whether the target page has public opinion information related to the financial institution, and N is a positive integer;

[0106] The second extraction unit 52 is used to extract features from the target text to obtain text features;

[0107] The processing unit 53 is used to input N pictures into a visual feature extractor and output target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from one picture, and the M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain target picture features, where M and T are positive integers;

[0108] A fusion unit 54 is used to fuse the text features and the target image features to obtain target fusion features;

[0109] The determination unit 55 is used to determine the information detection result based on the target fusion feature, wherein the information detection result is used to indicate whether the target page has public opinion information related to the financial institution.

[0110] In the information detection device provided in the second embodiment of the present invention, the first extraction unit 51 can respond to the information detection request, extract information from the target page, and obtain N pictures and target text, wherein the information detection request is used to request to detect whether the target page has public opinion information related to financial institutions, N is a positive integer, and the second extraction unit 52 extracts features of the target text to obtain text features, and the processing unit 53 inputs the N pictures into the visual feature extractor to output the target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from a picture, and the M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain the target picture features, M and T are positive integers, and the fusion unit 54 performs feature fusion on the text features and the target picture features to obtain the target fusion features, and the determination unit 55 determines the information detection result based on the target fusion features, wherein the information detection result is used to indicate whether the target page has public opinion information related to financial institutions. Thus, the technical problem of inaccurate detection results in the related art of using single-modal data for financial public opinion detection is solved.

[0111] In this embodiment, feature extraction is performed on the text features of the target text, feature extraction is performed on the image features of N pictures through M residual networks in the visual feature extractor, the text features and the image features of multiple pictures are fused, and financial public opinion information detection is performed based on the fused features, thereby avoiding the situation in the related technology of performing financial public opinion information detection based on a single picture or a single text and resulting in inaccurate detection results, thereby achieving the technical effect of improving the accuracy of financial public opinion information results.

[0112] Optionally, in the information detection device provided in the embodiment of the present application, the processing unit includes: a sorting subunit, used to sort N pictures according to preset sorting rules to obtain sorted pictures, wherein the preset sorting rules include: time order; a first processing subunit, used to input the sorted pictures into a visual feature extractor in sequence, and output target picture features.

[0113] Optionally, in the information detection device provided in the embodiment of the present application, the first processing sub-unit includes: a first processing module, which is used to, when N is less than M, sequentially supplement the sorted pictures starting from the first picture until the number of sorted pictures reaches M, thereby obtaining M sorted pictures; and a second processing module, which is used to input the sorted M pictures into a visual feature extractor in sequence, and output target picture features.

[0114] Optionally, in the information detection device provided in the embodiment of the present application, the second processing module includes: a first processing submodule, which is used to input the first image into the first residual network among the sorted M images, output the first image feature, input the second image into the second residual network, output the second image feature, and use the first first fully connected layer to perform feature fusion on the first image feature and the second image feature to obtain a first fused feature, wherein the dimensions of the first image feature, the second image feature and the first fused feature are the same; a second processing submodule, which is used to input the third image into the third residual network among the sorted M images, output the third image feature, and use the second first fully connected layer to perform feature fusion on the third image feature and the first fused feature to obtain a second fused feature, wherein the dimensions of the third image feature, the first fused feature and the second fused feature are the same; a third processing submodule, which is used to determine the target image feature based on the second fused feature, the fourth to last residual network, the fourth to last image in the sorted M images, and the third to last first fully connected layer, and output the target image feature through the last first fully connected layer.

[0115] Optionally, in the information detection device provided in the embodiment of the present application, the first extraction unit includes: a second processing sub-unit, used to determine the total playback time of the video when the target page includes the video, and determine the target time interval based on the ratio of the total playback time to N; an interception sub-unit, used to intercept the video according to the target time interval to obtain N pictures.

[0116] Optionally, in the information detection device provided in the embodiment of the present application, the dimensions of the text features, the target image features and the target fusion features are the same, and the fusion unit includes: a third processing subunit, used to input the text features and the target image features into the second fully connected layer; and output the target fusion features.

[0117] Optionally, in the information detection device provided in the embodiment of the present application, the determination unit includes: a third processing sub-unit, used to input the target fusion feature into the detector, and use the normalized exponential function in the detector to determine the information detection result, wherein the detector is used to detect whether there is public opinion information related to financial institutions in the target fusion feature.

[0118] It should be noted that the first extraction unit 51, the second extraction unit 52, the processing unit 53, the fusion unit 54 and the determination unit 55 correspond to steps S201 to S205 in the first embodiment, and the two modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above modules can also be run in the computer terminal 10 provided in the first embodiment as part of the device.

[0119] Embodiment 3

[0120] An embodiment of the present application may provide an electronic device, Figure 6 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, the electronic device may include: one or more ( Figure 6 (only one is shown) processor 602, memory 604, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0121] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: respond to the information detection request, extract information from the target page, and obtain N pictures and target text, wherein the information detection request is used to request to detect whether the target page has public opinion information related to the financial institution, and N is a positive integer; extract features from the target text to obtain text features; input the N pictures into the visual feature extractor, and output target picture features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each residual network is used to extract features from a picture, and the M first fully connected layers are used to perform feature fusion on the picture features extracted by the M residual networks to obtain target picture features, and M and T are positive integers; perform feature fusion on the text features and the target picture features to obtain target fusion features; based on the target fusion features, determine the information detection result, wherein the information detection result is used to indicate whether the target page has public opinion information related to the financial institution.

[0123] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: input N pictures into the visual feature extractor and output the target picture features, including: sorting the N pictures according to preset sorting rules to obtain sorted pictures, wherein the preset sorting rules include: time order; inputting the sorted pictures into the visual feature extractor in sequence and outputting the target picture features.

[0124] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: input the sorted pictures into the visual feature extractor in sequence, and output the target picture features, including: when N is less than M, in the sorted pictures, starting from the first picture, supplementing the sorted pictures in sequence until the number of sorted pictures reaches M, to obtain M sorted pictures; input the sorted M pictures into the visual feature extractor in sequence, and output the target picture features.

[0125] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: input the sorted M pictures into the visual feature extractor in sequence and output the target picture feature, including: among the sorted M pictures, input the first picture into the first residual network to output the first picture feature, input the second picture into the second residual network to output the second picture feature, and use the first first fully connected layer to perform feature fusion on the first picture feature and the second picture feature to obtain a first fusion feature, wherein the dimensions of the first picture feature, the second picture feature and the first fusion feature are the same; among the sorted M pictures, input the third picture into the third residual network to output the third picture feature, and use the second first fully connected layer to perform feature fusion on the third picture feature and the first fusion feature to obtain a second fusion feature, wherein the dimensions of the third picture feature, the first fusion feature and the second fusion feature are the same; based on the second fusion feature, the fourth to the last residual network, the fourth to the last picture in the sorted M pictures, and the third to the last first fully connected layer, determine the target picture feature, and output the target picture feature through the last first fully connected layer.

[0126] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: extract information from the target page to obtain N pictures, including: when the target page includes a video, determine the total playback time of the video, and determine the target time interval based on the ratio of the total playback time to N; intercept the video according to the target time interval to obtain N pictures.

[0127] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: the dimensions of the text features, the target image features and the target fusion features are the same, and the text features and the target image features are fused to obtain the target fusion features, including: inputting the text features and the target image features into the second fully connected layer; and outputting the target fusion features.

[0128] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: determine the information detection results based on the target fusion features, including: inputting the target fusion features into the detector, and using the normalized exponential function in the detector to determine the information detection results, wherein the detector is used to detect whether there is public opinion information related to financial institutions in the target fusion features.

[0129] By adopting the embodiment of the present application, a scheme for information detection is provided, which extracts the text features of the target text, extracts the image features of N pictures through M residual networks in the visual feature extractor, fuses the text features and the image features of multiple pictures, and detects financial public opinion information based on the fused features, thus avoiding the situation in the related art where financial public opinion information detection is performed based on a single picture or a single text, resulting in inaccurate detection results, thereby achieving the technical effect of improving the accuracy of financial public opinion information results. This solves the technical problem in the related art where single-modal data is used for financial public opinion detection, resulting in inaccurate detection results.

[0130] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device may also be a smart phone, a tablet computer, a PDA, a mobile Internet device (MID), a PAD or other terminal device. Figure 6 The structure of the electronic device is not limited. Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 6 Different configurations are shown.

[0131] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0132] Embodiment 4

[0133] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the information detection method provided in the first embodiment.

[0134] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0135] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the program steps of the information detection method.

[0136] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0137] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0142] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An information detection method, characterized in that: include: In response to the information detection request, information is extracted from the target page to obtain N pictures and target texts, wherein the information detection request is used to request to detect whether the target page has public opinion information related to the financial institution, and N is a positive integer; Extracting features from the target text to obtain text features; Inputting N of the images into a visual feature extractor, and outputting target image features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each of the residual networks is used to extract features from one of the images, and the M first fully connected layers are used to perform feature fusion on the image features extracted by the M residual networks to obtain the target image features, where M and T are positive integers; Performing feature fusion on the text feature and the target image feature to obtain a target fusion feature; Based on the target fusion feature, an information detection result is determined, wherein the information detection result is used to indicate whether the target page has public opinion information related to a financial institution.

2. The information detection method according to claim 1, characterized in that: Input N of the above pictures into the visual feature extractor, and output the target picture features, including: Sorting the N pictures according to a preset sorting rule to obtain sorted pictures, wherein the preset sorting rule includes: time order; The sorted images are sequentially input into the visual feature extractor, and the target image features are output.

3. The information detection method according to claim 2, characterized in that: Inputting the sorted pictures into the visual feature extractor in sequence, and outputting the target picture features, including: When N is less than M, in the sorted pictures, starting from the first picture, the pictures after the sorting are supplemented in order until the number of sorted pictures reaches M, thereby obtaining M sorted pictures; The sorted M pictures are sequentially input into the visual feature extractor, and the target picture features are output.

4. The information detection method according to claim 3, characterized in that: Inputting the sorted M pictures into the visual feature extractor in sequence, and outputting the target picture features, including: Among the sorted M pictures, input the first picture into the first residual network to output a first picture feature, input the second picture into the second residual network to output a second picture feature, and use the first first fully connected layer to perform feature fusion on the first picture feature and the second picture feature to obtain a first fused feature, wherein the dimensions of the first picture feature, the second picture feature and the first fused feature are the same; Among the sorted M pictures, input the third picture into the third residual network, output the third picture feature, and use the second first fully connected layer to perform feature fusion on the third picture feature and the first fusion feature to obtain a second fusion feature, wherein the dimensions of the third picture feature, the first fusion feature and the second fusion feature are the same; Based on the second fusion feature, the fourth to last residual network, the fourth to last picture in the sorted M pictures, and the third to last first fully connected layer, determine the target picture feature, and output the target picture feature through the last first fully connected layer.

5. The information detection method according to claim 1, characterized in that: Extract information from the target page and obtain N pictures, including: In the case where the target page includes a video, determining a total playback time of the video, and determining a target time interval based on a ratio of the total playback time to N; The video is intercepted according to the target time interval to obtain N pictures.

6. The information detection method according to claim 1, characterized in that: The dimensions of the text features, the target image features and the target fusion features are the same, and the text features and the target image features are fused to obtain the target fusion features, including: inputting the text features and the target image features into a second fully connected layer; and outputting the target fusion features.

7. The information detection method according to claim 1, characterized in that: Based on the target fusion feature, determining the information detection result includes: inputting the target fusion feature into a detector, and using a normalized exponential function in the detector to determine the information detection result, wherein the detector is used to detect whether there is public opinion information related to the financial institution in the target fusion feature.

8. An information detection device, characterized in that: include: A first extraction unit is used to respond to an information detection request, extract information from a target page, and obtain N pictures and target texts, wherein the information detection request is used to request to detect whether the target page has public opinion information related to a financial institution, and N is a positive integer; A second extraction unit is used to extract features from the target text to obtain text features; A processing unit, configured to input N of the images into a visual feature extractor, and output target image features, wherein the visual feature extractor includes: M residual networks and T first fully connected layers, each of the residual networks is configured to perform feature extraction on one of the images, and the M first fully connected layers are configured to perform feature fusion on the image features extracted by the M residual networks to obtain the target image features, and M and T are positive integers; A fusion unit, used for fusing the text feature and the target image feature to obtain a target fusion feature; A determination unit is used to determine an information detection result based on the target fusion feature, wherein the information detection result is used to indicate whether the target page has public opinion information related to a financial institution.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the information detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein the program executes the information detection method described in any one of claims 1 to 7 when running.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the information detection method described in any one of claims 1 to 7 are implemented.

Images