A method and apparatus for extracting text content

Abstract

The embodiment of the present application discloses a kind of text content extraction method and device.According to the embodiment of the present application, the image features of the text segment in at least two images are extracted, the similarity between the image features of the text segment in the image is determined by similarity calculation on the similar parts of the text segment in the image, and the text segment is merged according to the similarity, so as to obtain the corresponding text content of the at least two images.Because the similarity between the image features is determined by similarity calculation on the similar parts of the text segment in at least two images, that is, the alignment processing is done after dynamic matching of the text segment in at least two images, the similar parts are matched and then similarity calculation is performed, compared with the scheme of directly similarity calculation on the text segment, the obtained similarity more accurately represents the similarity between the text segments, without labeling the sequence characteristics between the text segments, the image text segment can be better associated.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a text content extraction method and device, an image information analysis network processing method and device, an image retrieval method and device, a course content extraction method and device, a subtitle extraction method and device, and corresponding electronic equipment and machine-readable medium. BACKGROUND

[0002] With the rapid popularization of personal-oriented consumer electronics products such as mobile phones and tablet computers, and the rapid development of social media and video platforms such as long videos, short videos, and live broadcasts, more and more users begin to use video media for daily activities such as knowledge learning, drama watching, and news browsing.

[0003] A video usually contains a large amount of text information in the form of post-subtitles, scene text, etc. The association and extraction of video text are very important in different scenarios. If the text association technology is not used or the effect of the text association technology is poor, the video content extraction result will have poor relevance in content and time, and the text information provided to the client is scattered and disordered. Therefore, the pros and cons of the association effect directly affect the user experience of using the video product.

[0004] With the diversification of video application scenarios, the text between video frames changes greatly in form and content. However, the existing text segment association scheme based on image features of text segments has poor effect when applied to scenarios where the form and content of text segments change greatly. SUMMARY

[0005] In view of the above problems, the present application is proposed to provide a text content extraction, image information analysis network processing, image retrieval, course content extraction, and subtitle extraction method and device, and computer equipment and machine-readable medium that overcome the above problems or at least partially solve the above problems.

[0006] According to one aspect of the present application, a text content extraction method is provided, comprising:

[0007] extracting image features of text segments in at least two images;

[0008] determining the similarity between the image features of the text segments in the at least two images by performing similarity calculation on similar parts of the text segments in the at least two images;

[0009] merging the text segments in the at least two images according to the similarity to obtain the text content corresponding to the at least two images.

[0010] According to another aspect of the present application, a processing method of an image information parsing network is provided, comprising:

[0011] obtaining a pair of associated image samples;

[0012] extracting image features of text segments in the pair of associated image samples respectively;

[0013] determining similarities between the image features of the text segments in the pair of associated image samples by performing similarity calculation on similar parts of the text segments in the pair of associated image samples;

[0014] training the image information parsing network according to the similarities determined for the pair of associated image samples, for extracting image features of text segments in images.

[0015] According to another aspect of the present application, a processing method of an image information parsing network is provided, comprising:

[0016] obtaining a triple sample group comprising a pair of associated image samples and a pair of non-associated image samples;

[0017] extracting image features of text segments in the triple sample group respectively;

[0018] determining similarities between the image features of the text segments in the pair of associated image samples by performing similarity calculation on similar parts of the text segments in the pair of associated image samples;

[0019] determining similarities between the image features of the text segments in the pair of non-associated image samples by performing similarity calculation on similar parts of the text segments in the pair of non-associated image samples;

[0020] training the image information parsing network according to the similarities determined for the pair of associated image samples and the similarities determined for the pair of non-associated image samples, for extracting image features of text segments in images.

[0021] According to another aspect of the present application, an image retrieval method is provided, comprising:

[0022] obtaining a first image for retrieval;

[0023] extracting image features of text segments in the first image;

[0024] determining similarities between the image features of the text segments in the first image and text segments in a second image by performing similarity calculation on similar parts of the text segments respectively included in the first image and the second image;

[0025] determining the second image as an associated image of the first image according to the similarities;

[0026] The associated image is provided as an image search result.

[0027] According to another aspect of the present application, a course content extraction method is provided, comprising:

[0028] extracting image features of text segments in image frames of a course video;

[0029] determining similarity between image features of text segments corresponding to two adjacent image frames by similarity calculation on similar parts of the text segments in the two adjacent image frames;

[0030] merging text segments in the two adjacent image frames with similarity satisfying a set range, to obtain text content corresponding to the course video.

[0031] According to another aspect of the present application, a subtitle extraction method is provided, comprising:

[0032] extracting image features of text segments in image frames of a target video;

[0033] determining similarity between image features of text segments corresponding to two adjacent image frames by similarity calculation on similar parts of the text segments in the two adjacent image frames;

[0034] merging text segments in the two adjacent image frames with similarity satisfying a set range, to obtain text content corresponding to the course video.

[0035] According to another aspect of the present application, a video content extraction method is provided, comprising:

[0036] extracting image features of text segments in image frames of a target video;

[0037] determining similarity between image features of text segments in two adjacent image frames by similarity calculation on similar parts of the text segments in the two adjacent image frames;

[0038] merging text segments in the two adjacent image frames according to the similarity, to obtain text content corresponding to the target video.

[0039] According to another aspect of the present application, a conference content processing method is provided, comprising:

[0040] real-time acquiring a conference video;

[0041] extracting image features of text segments in image frames of a conference video;

[0042] The similarity between the image features of the text segments corresponding to the two adjacent image frames is determined by performing similarity calculation on similar parts of the text segments in the two adjacent image frames.

[0043] The text segments in the two adjacent image frames whose similarity meets a set range are merged.

[0044] The text content obtained after the merging is added to the conference video as a subtitle.

[0045] The conference video with the added subtitle is provided.

[0046] According to another aspect of the present application, a remote video processing method is provided, comprising:

[0047] The image features of text segments in image frames of a remote medical video are identified.

[0048] The similarity between the image features of the text segments corresponding to the two adjacent image frames is determined by performing similarity calculation on similar parts of the text segments in the two adjacent image frames.

[0049] The text segments in the two adjacent image frames whose similarity meets a set range are merged.

[0050] The text content obtained after the merging is added to the conference video as a subtitle.

[0051] The conference video with the added subtitle is provided.

[0052] According to another aspect of the present application, a text content extraction method is provided, comprising:

[0053] At least two images are obtained.

[0054] The image features of text segments in the at least two images are extracted.

[0055] The similarity between the image features of the text segments in the at least two images is determined by performing similarity calculation on similar parts of the text segments.

[0056] The text segments in the at least two images are merged according to the similarity, and text content corresponding to the at least two images is obtained.

[0057] The text content is provided.

[0058] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and

[0059] a memory having stored thereon executable code that, when executed, is to cause the processor to perform the method of any of the above.

[0060] According to another aspect of the present application, one or more machine readable media have stored thereon executable code that, when executed, is to cause a processor to perform the method of any of the above.

[0061] According to an embodiment of the present application, after the image features of the text segments in the at least two images are extracted, the similarity between the image features of the text segments in the at least two images is determined by performing similarity calculation on similar parts of the text segments in the at least two images, and the text segments are further processed by merging according to the similarity, so as to obtain the text content corresponding to the at least two images. Since the similarity between the image features is determined by performing similarity calculation on the similar parts of the text segments in the at least two images, that is, the alignment processing is performed after the dynamic matching of the text segments in the at least two images, and the similarity calculation is performed after the corresponding of the similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, without the need of labeling the sequence characteristics between the text segments, the image text segments can be better associated, the extraction result of the video content is simple and complete, the requirement of the output content is met, and the use experience of the user is improved. Moreover, the sequence characteristics of the partially overlapped text segments between the video frames in the video scene can be better adapted to, and the universality is stronger.

[0062] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application, the content of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0063] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to depict only preferred embodiments of the application, and therefore should not be considered to limit the scope of the application in any way. Furthermore, the drawings are merely intended to show certain embodiments of the application, and should not be considered limiting the scope of the application in any way. In the drawings:

[0064] Figure 1 An example of a video content extraction method according to an embodiment of the present application is shown;

[0065] Figure 2 A flowchart of a text content extraction method according to an embodiment of the present application is shown;

[0066] Figure 3A flow chart of a processing method of an image information analysis network according to Embodiment Two of the present application is shown;

[0067] Figure 4 A flow chart of a processing method of an image information analysis network according to Embodiment Three of the present application is shown;

[0068] Figure 5 A flow chart of an image retrieval method according to Embodiment Four of the present application is shown;

[0069] Figure 6 A flow chart of a course content extraction method according to Embodiment Five of the present application is shown;

[0070] Figure 7 A flow chart of a subtitle extraction method according to Embodiment Six of the present application is shown;

[0071] Figure 8 A flow chart of a video content extraction method according to Embodiment Seven of the present application is shown;

[0072] Figure 9 A flow chart of a conference content processing method according to Embodiment Eight of the present application is shown;

[0073] Figure 10 A flow chart of a remote video processing method according to Embodiment Nine of the present application is shown;

[0074] Figure 11 A flow chart of a text content extraction method according to Embodiment Ten of the present application is shown;

[0075] Figure 12 A structural block diagram of a text content extraction device according to Embodiment Eleven of the present application is shown;

[0076] Figure 13 A structural block diagram of a processing device of an image information analysis network according to Embodiment Twelve of the present application is shown;

[0077] Figure 14 A structural block diagram of a processing device of an image information analysis network according to Embodiment Thirteen of the present application is shown;

[0078] Figure 15 A structural block diagram of an image retrieval device according to Embodiment Fourteen of the present application is shown;

[0079] Figure 16 A structural block diagram of a course content extraction device according to Embodiment Fifteen of the present application is shown;

[0080] Figure 17 A structural block diagram of a subtitle extraction device according to Embodiment Sixteen of the present application is shown;

[0081] Figure 18 Fig. 17 shows a structural block diagram of a video content extraction device according to an embodiment of the present application;

[0082] Figure 19 Fig. 18 shows a structural block diagram of a conference content processing device according to an embodiment of the present application;

[0083] Figure 20 Fig. 19 shows a structural block diagram of a remote video processing device according to an embodiment of the present application;

[0084] Figure 21 Fig. 20 shows a structural block diagram of a text content extraction device according to an embodiment of the present application;

[0085] Figure 22 Fig. 21 shows an interaction schematic diagram of a terminal and a cloud service platform;

[0086] Figure 23 Fig. 22 shows an exemplary system that can be used to implement various embodiments described in the present disclosure. DETAILED DESCRIPTION

[0087] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0088] Existing video-text association algorithms can be mainly divided into association algorithms based on the position relationship of adjacent frames of video text, association algorithms based on the similarity relationship of text line image features, and the fusion of the two methods.

[0089] The text association algorithm based on the position relationship of text is an area intersection over union scheme, that is, whether the ratio of the overlapping area of two text lines of the current frame and the previous frame to the total coverage area is greater than a certain threshold is calculated to determine whether the text of the previous and next frames is in an association relationship. Since only the characteristics of the text content itself are considered, the recognition effect of the text with display effect is poor.

[0090] The method based on traditional image features extracts the pre-designed related display features such as brightness and background of the text line. Compared with the text association algorithm, it can better adapt to the scene with text display effect, but it is sensitive to changes in lighting, text occlusion, and similar backgrounds, so the effect is not good in scene generalization.

[0091] Especially for the scene where the text segment form and content change greatly, the text segment association and video content extraction are not good.

[0092] Therefore, the embodiment of the present application proposes a new text content extraction scheme, and gives the corresponding image information analysis network processing, image retrieval, course content extraction, subtitle extraction and other related application schemes.

[0093] In the text content extraction scheme of the embodiment of the present application, the similarity of the text segments is judged based on the image features of the text segments in the image, and the similar text segments are determined as the associated text segments, and the text content of the image is integrated according to the associated text segments.

[0094] The above text segment can be a text line, that is, a plurality of texts listed in rows, or a plurality of texts in other forms.

[0095] The scheme of the embodiment of the present application can be applied to the text content extraction between two or more images, which is described below taking two images as an example.

[0096] The similarity between the image features of the text segments is determined by the similar part of the text segments, that is, the embodiment of the present application distinguishes the similar part and the non-similar part between the text segments of the two images when calculating the similarity of the image features, for example, the similar part is calculated for similarity, and the other part is calculated for similarity. The similarity calculation results of different parts can be processed into the similarity between the image features of the text segments in the associated image sample pair by summation, weighted average and the like. The similar part here can be divided into a plurality of sub-features in the feature space width of the text segment, which can be divided into a plurality of sub-features with the same length or different lengths. The sub-features corresponding to the two images are compared to find the similar part, and the similar part can have a similarity that meets a certain numerical range or a higher similarity than other parts.

[0097] The feature space of the image feature usually has three attributes of width, height and feature channel. The above division in the feature space width can determine the number of divisions and the length of a single sub-feature according to actual needs.

[0098] For example, the text segments extracted from the two images are "happy new" and "new year" respectively, the image regions corresponding to the two text segments can be divided into image features corresponding to three characters as a group, that is, "happy new" is divided into "hap", "py" and "new", and "new year" is divided into "new", "ye" and "ar", wherein "new" is a similar part of the two associated image samples. If the prior art scheme is followed, the similarity between the three groups of sub-features is calculated in sequence, that is, the similarity between "hap" and "new", "py" and "ye", and "new" and "ar" is calculated respectively. According to the scheme of the present application, "new" belongs to the similar part of the two samples, and the similarity calculation is performed thereon. When other similarity calculations are performed, they can be random or selected according to the corresponding positions.

[0099] According to the embodiment of the present application, after extracting the image features of the text segments in the at least two images, the similarity between the image features of the text segments in the at least two images is determined by similarity calculation on the similar parts of the text segments in the at least two images. Further, the text segments are merged according to the similarity, so as to obtain the text content corresponding to the at least two images. Since the similarity between the image features is determined by similarity calculation on the similar parts of the text segments in the at least two images, that is, the alignment processing is performed after dynamic matching of the text segments in the at least two images, and the similarity calculation is performed after the corresponding similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments. Without labeling the sequence characteristics between the text segments, the image text segments can be better associated, so that the extraction result of the video content is simple and complete, the requirement of the output content is met, and the user's use experience is improved. Moreover, it can better adapt to the sequence characteristics of the partially overlapping text segments between the video frames in the video scene, and has stronger universality.

[0100] If the similarity meets the set condition, it means that the text segments have the same content or similar content, and the two text segments are associated text segments. At this time, the text segments in the two images can be merged to obtain the text content corresponding to the two images. The similarity meeting the set condition can be that the similarity exceeds the set threshold, or the similarity represented by the similarity is ranked in the front, and the text segments of the two images are determined as associated text segments.

[0101] The merging processing here can be a deduplication processing or a completion processing on the repeated text segments. For the case that the subtitles in the video have overlapping parts in two images, the deduplication can be performed, for example, the two images correspond to "happy new year" and "happy new year to you all" respectively, and "happy new year to you all" is obtained after the deduplication. For the case that the text segments in a single image are not completely displayed, the completion can be performed, that is, the local text is integrated in time sequence, for example, the two images correspond to "happy new" and "year to you all" respectively, and "happy new year to you all" is obtained after the completion. The two processing can also be performed simultaneously, for example, the two images correspond to "happy new" and "new year" respectively, and "happy new year" is obtained after the deduplication and completion.

[0102] Through the above filtering of the repeated text content and the integration of the local text in time sequence, the generated video content is concise and complete, which meets the requirement of the output content and greatly improves the user experience.

[0103] In an optional embodiment, the obtained text content after the merging can be provided on the device interface, and the associated editing control is provided. The device user (for example, a video subtitle maker) can edit the text content based on the editing control. Based on the editing operation on the text content, the updated text content can be obtained, and the updated text content is further displayed on the device interface. Alternatively, the text content can be confirmed, and the confirmed result is displayed on the device interface, or the text segment in the image is updated.

[0104] The image feature extraction above is a concept in computer vision and image processing, which refers to the information of an image extracted by a computer in one or more set dimensions. In the embodiment of the present application, the image feature of the text segment is the image feature of the image region where the text segment is located, which can include the image feature presented by the content of the text segment, for example, can include color feature, texture feature, shape feature and spatial relationship feature. The color feature and the texture feature describe the surface properties of the scene corresponding to the image or the image region. The shape feature can be the contour feature of the outer boundary of an object, or a region feature representing the information of the region. The spatial relationship feature refers to the spatial position relationship between multiple objects in the image.

[0105] The image feature can represent an image or a partial region in the image, and thus can be used to replace the image or the region itself to perform various comparisons or operations. For example, for text segments in at least two images, if the image features corresponding to the text segments are similar to each other and meet certain conditions, it is determined that the two text segments are similar.

[0106] The extracted features in the application can be distributed in multiple dimensions, and the application does not limit this. For example, low-level features such as brightness and color histogram can be included, which have good robustness to text lines with angle and scale changes in a video. High-level features such as scale invariant feature transform (SIFT) and speeded up robust features (SURF) can also be included, which have good recognition stability to changes in conditions such as illumination, translation, and rotation. In order to adapt to the scene where the fast movement of the lens or the video scene may cause image blur, the random sample consensus (RANSAC) algorithm can also be used to filter feature mismatch and constrain geometric registration.

[0107] In an optional embodiment, when extracting the image feature of the text segment in the image frame of the target video, the image region where the text segment is located can be divided into multiple character regions, the image features of the character regions are extracted respectively, and the image features of the character regions are combined as the image feature of the text segment. Compared with the scheme of using only a single character region, the image features of the character regions are retained by segmenting multiple character regions, so that the resolution of the image feature in the spatial width is retained, and the extracted image feature can more accurately represent the text segment. Specifically, the extracted image feature can be represented as a matrix [w, k], where w represents the spatial width of the feature, and k represents the number of channels of the image feature, that is, the number of dimensions of the image feature.

[0108] The image region can be divided according to the number of character regions set in advance. For example, if the image region is divided into four character regions, the spatial width of the image feature is four units. The image features at different positions in a single character region can be aggregated and counted by a pooling operation to obtain the image feature in the character region. The output image feature can also be normalized to the same representation scale.

[0109] The above similarity calculation can be implemented by any similarity calculation algorithm. For example, the Euclidean distance, the cosine distance, the Manhattan distance (L1 norm distance), or the like can be used to calculate the distance between the image features as the similarity between the image features.

[0110] In an optional embodiment, when determining the similarity between the image features of the text segments in the at least two images by performing similarity calculation on the similar parts of the text segments in the at least two images, the similarity between the sub-features of the text segments of the pair of associated image samples can be determined according to the plurality of sub-features divided by the text segments in the feature space width, that is, the similarity between all the sub-features corresponding to the pair of associated image samples is calculated two by two. Further, the required correspondence is selected from all the similarities calculated above, that is, the sub-features including the similar parts are associated according to the similarity between the sub-features, for example, according to the size or order of the similarity, two sub-features with a similarity meeting the requirements can be determined as the sub-features including the similar parts for association.

[0111] According to the correspondence of the associated sub-features, the similarity between the image features of the text segments in the pair of associated image samples is determined. For example, the sub-features including the similar parts determined according to the similarity are used to obtain the calculated similarity, the similarity of the remaining sub-features is calculated, and then the similarities calculated for each sub-feature are summed or weighted, etc., to obtain the similarity corresponding to the two text segments.

[0112] In an optional embodiment, when associating the sub-features including the similar parts according to the similarity between the sub-features, the minimum path method can be used to construct a similarity matrix according to the similarity between all the sub-features of the text segments of the pair of associated image samples, determine the minimum cost path of the similarity matrix, and use the correspondence of the sub-features represented by the minimum cost path as the correspondence of the associated sub-features. Since the similarity is represented by the distance between the image features, the sum of the similarities in the minimum cost path is the smallest, that is, the distance between the image features of all the sub-features is the smallest, and the two sub-features corresponding to each similarity in the minimum cost path are the most similar sub-features selected from all the sub-features for similarity calculation.

[0113] The minimum cost path can be determined using dynamic programming, greedy algorithm, Dijkstra algorithm (single-source shortest path algorithm), Bellman-Ford algorithm, etc. Taking the dynamic programming algorithm as an example, based on the basic idea of mutual dependence between optimal substructures, all possible solutions are enumerated and traversed, and optimization calculation is performed at each path sub-node, so as to find the optimal solution.

[0114] Further, in the correspondence relationship between the associated sub-features, the similarity between the image features of the text segments in the associated image sample pair is determined. Since the most similar sub-features on the minimum cost path are calculated for similarity, the similarity between the image features of the text segments in the associated image sample pair can be determined according to the similarity of the sub-features on the minimum cost path. According to the above discussion, compared with the scheme of directly calculating the similarity of the text segments, the obtained similarity more accurately represents the similarity between the text segments.

[0115] In an optional embodiment, the image features of the text segments in the image frame can be extracted based on an image information analysis network. The image information analysis network is used to extract the image features of the text segments, and the image information analysis network is trained by the associated image sample pairs collected in advance.

[0116] The associated image sample pair includes two image samples associated with the text segments. The collected associated image sample pair includes multiple groups. The image information analysis network is trained according to the similarity between the image features of the text segments in the associated image sample pair, so that the similarity calculated according to the image features extracted from the associated image sample pair by the image information analysis network conforms to the similarity characteristics of the associated image sample pair.

[0117] Compared with the existing scheme of extracting pre-designed image features, the embodiments of the present application can use a deep learning algorithm to train the image information analysis network, which can better adaptively learn the image features under projection transformation, light shielding, jitter blur, and the like, and can be applicable to the case where the shape and content of the text segments change greatly in the video process, and can more accurately associate the text segments in the video frames.

[0118] The image information analysis network mentioned above can be a convolutional neural network (CNN, Convolutional Neural Networks), and can also be other self-learning neural networks, such as a recurrent neural network (RNN, Recurrent Neural Network) and the like.

[0119] Before the above scheme, the image information analysis network can also be pre-trained. Specifically, the associated image sample pair can be obtained, and the image information analysis network used to extract the image features of the text segments in the image can be trained according to the associated image sample pair.

[0120] In an optional embodiment, image features of the text segments in the associated image sample pair can be extracted respectively, similarity between the image features of the text segments in the associated image sample pair is determined by performing similarity calculation on similar parts of the text segments in the associated image sample pair, and the image information analysis network is further trained according to the similarity determined for the associated image sample pair, for extracting image features of text segments in an image.

[0121] In an optional embodiment, when the associated image sample pair is obtained, text segments (tracket, i.e., a plurality of frames of text segments) can be extracted from image frames of the video sample respectively, and then image frames of the text segments corresponding to similar parts are taken as the associated image sample pair, i.e., the positive sample pair.

[0122] In addition, image frames of text segments without similar parts can also be taken as the non-associated image sample pair, i.e., the negative sample pair, combined with the positive sample pair, i.e., a ternary sample group including the positive sample pair and the negative sample pair can be constructed, for example, the ternary sample group includes three samples A, B and C, A and B are the positive sample pair, and B and C or A and C form the negative sample pair.

[0123] For example, text segments are extracted from three continuous image frames of a video, which are “ricomer”, “br mer” and “ALL” respectively, after similarity calculation by extracting image features of the text segments, it is determined that the first two text segments include similar contents r and mer, and the first two text segments do not include similar contents with the last text segment, so the first two image frames can be taken as the associated image sample pair, and the first two image frames can be taken as the non-associated image sample pair with the third image frame.

[0124] Further, similarity between the image features of the text segments in the non-associated image sample pair is determined by performing similarity calculation on similar parts of the text segments in the non-associated image sample pair. Thus, when the image information analysis network is trained according to the positive sample pair and the negative sample pair, the image information analysis network is trained according to the similarity determined for the associated image sample pair and the similarity determined for the non-associated image sample pair, i.e., the training direction of the image information analysis network is to make the similarity of the positive sample pair higher and the similarity of the negative sample pair lower, so that the image information analysis network with more accurate representation can be obtained.

[0125] In an optional embodiment, after the image information analysis network outputs the image features, the dynamic matching and alignment scheme similar to the above can be continued to be used when determining the similarity between the image features of the text segments in the associated image sample pair, by performing similarity calculation on similar parts of the text segments in adjacent two image frames, the similarity between the image features of the text segments in the adjacent two image frames is determined.

[0126] Correspondingly, when determining the similarity between the image features of the text segments in the pair of associated image samples by performing similarity calculation on the similar parts of the text segments in the pair of associated image samples, the similarity between the sub-features of the text segments in the pair of associated image samples can be determined according to the plurality of sub-features included in the text segments, and then the sub-features including the similar parts are associated according to the similarity between the sub-features, and then the similarity between the image features of the text segments in the pair of associated image samples is determined according to the correspondence relationship of the associated sub-features.

[0127] In an optional embodiment, when associating the sub-features including the similar parts according to the similarity between the sub-features, a similarity matrix can be constructed according to the similarity between all sub-features of the text segments in the pair of associated image samples, and the minimum cost path of the similarity matrix is determined by dynamic programming, and the correspondence relationship of the sub-features represented by the minimum cost path is taken as the correspondence relationship of the associated sub-features.

[0128] Correspondingly, when determining the similarity between the image features of the text segments in the pair of associated image samples according to the correspondence relationship of the associated sub-features, the similarity between the image features of the text segments in the pair of associated image samples can be determined according to the similarity of the sub-features on the minimum cost path.

[0129] It should be noted that the scheme of the present application can be implemented as an application, a service, an instance, a functional module in the form of software, a virtual machine (VM) or a container, or can also be implemented as a hardware device (such as a server or a terminal device) or a hardware chip (such as a CPU, a GPU or an FPGA) with image processing function.

[0130] The cloud service platform can provide corresponding services for videos or images using its own computing resources. End users (such as subtitle / video / image editing professionals or ordinary users) can obtain processing services for videos or images through a client or a setting interface. After the end users obtain the processing results of the cloud service platform for the text segments in the videos or images, they can further perform personalized editing as needed.

[0131] Alternatively, the cloud service platform can provide training functions of the image information parsing network using its own computing resources. The video platform can apply for training of the image information parsing network through a client or a setting interface, and can also submit video samples for extracting a triple sample set. The cloud service platform trains the image information parsing network according to the triple sample set by the above method, and provides it to the video platform. The video platform further uses the image information parsing network for subsequent extraction of video content, extraction of course content, extraction of subtitles and other related fields.

[0132] The embodiments of the present application can be applied to extraction of video content in various application scenarios, for example, processing of text segments in education live broadcast, entertainment live broadcast, or remote course video, news video, street view video, commodity video, and entertainment video, such as animation text of variety show, scrolling text in news program, signboard information in street view video, commodity information of e-commerce live broadcast, and packaging text input of cylindrical commodity such as bottle and can, which can provide concise and complete information for convenient viewing or subsequent production.

[0133] The video content extraction can also be applied to processing of text segments in a video conference scenario to facilitate timely obtaining of key information in a conference video for viewing by a user participating in the video conference or after the conference. The video content extraction is applied to a remote video scenario, and text segments of multiple video frames are extracted and processed according to association therebetween to facilitate timely obtaining of medical key information for viewing by a user participating in the remote medical video or after the video.

[0134] In addition to being applied to extraction of video content, the present application can determine the appearance and disappearance time of a certain text information in a video according to association between text segments of video frames to mark the appearance position of the information on a time axis for subsequent positioning of the text information according to a time period.

[0135] Reference Figure 1 An example of a text content extraction method of the present application is shown. Three text segments are used to construct a triple sample group for training of an image information parsing network, in which the anchor is used as a basic sample, the positive is a relative positive sample, and the negative is a relative negative sample. In training of the image information parsing network, the widths of the anchor sample and the positive sample are extracted as a multi-dimensional matrix including four sub-features, the four sub-features corresponding to the anchor sample and the positive sample are calculated for similarity in pairs to obtain a four-dimensional matrix constructed by similarity, a minimum cost path of the four-dimensional matrix is found as shown in the figure, the sub-features corresponding to each similarity on the path are sub-features including similar parts, which are corresponded in pairs, and the similarity between the image features of the entire text segment is calculated, and the similarity is used as a value of a first loss function. Since the anchor sample and the negative sample have poor similarity, a multi-dimensional matrix with a width of one sub-feature can be constructed here, and the corresponding similarity is used as a value of a second loss function. The sum of the two loss functions is used as a total loss function, and the image information parsing network is iteratively trained.

[0136] Figure 1An example of extracting video content is also given, two text segments of "happy new" and "happy new year" are extracted from two video frames of a video, image features of the text segments are extracted by inputting image information into the image information parsing network, similarity is calculated according to the image features, and the text segments are determined to be associated text segments according to the similarity satisfying a condition, and the associated text segments are merged to obtain the extracted content "happy new year" of the video.

[0137] Referring to Figure 2 , a flowchart of a text content extraction method according to an embodiment of the present application is shown, which can specifically include the following steps:

[0138] Step 101, extracting image features of text segments in at least two images;

[0139] Step 102, determining similarity between image features of text segments in the at least two images by similarity calculation on similar parts of the text segments in the at least two images;

[0140] Step 103, merging the text segments in the at least two images according to the similarity to obtain text content corresponding to the at least two images.

[0141] According to the embodiment of the present application, after extracting image features of text segments in at least two images, similarity between image features of text segments in the at least two images is determined by similarity calculation on similar parts of the text segments in the at least two images, and the text segments are further merged according to the similarity, so as to obtain text content corresponding to the at least two images. Since the similarity between the image features is determined by similarity calculation on the similar parts of the text segments in the at least two images, that is, the alignment processing is performed after dynamic matching of the text segments in the at least two images, and the similarity calculation is performed after the corresponding similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, without labeling the sequence characteristics between the text segments, the image text segments can be better associated, the extraction result of the video content is simple and complete, the requirement of the output content is met, and the use experience of the user is improved. Moreover, it can better adapt to the sequence characteristics of the partially overlapping text segments between the video frames in the video scene, and has stronger universality.

[0142] In an optional embodiment, the extracting image features of text segments in at least two images includes:

[0143] dividing an image region where the text segment is located into a plurality of character regions;

[0144] extract image features of the character regions respectively, and combine the image features of the character regions into image features of the text segment.

[0145] In an alternative embodiment, the method further comprises:

[0146] obtaining a pair of associated image samples, and training an image information analysis network for extracting image features of a text segment in an image according to the pair of associated image samples.

[0147] In an alternative embodiment, the training of the image information analysis network for extracting image features of a text segment in an image according to the pair of associated image samples comprises:

[0148] extracting image features of the text segments in the pair of associated image samples respectively;

[0149] determining the similarity between the image features of the text segments in the pair of associated image samples by similarity calculation on similar parts of the text segments in the pair of associated image samples;

[0150] training the image information analysis network according to the similarity determined for the pair of associated image samples, for extracting image features of a text segment in an image.

[0151] In an alternative embodiment, the determination of the similarity between the image features of the text segments in the pair of associated image samples by similarity calculation on similar parts of the text segments in the pair of associated image samples comprises:

[0152] dividing the image features into a plurality of sub-features in the width of the feature space, and determining the similarity between the sub-features of the text segments of the at least two images;

[0153] associating the sub-features including the similar parts according to the similarity between the sub-features;

[0154] determining the similarity between the image features of the text segments in the at least two images according to the corresponding relationship of the associated sub-features.

[0155] In an alternative embodiment, the association of the sub-features including the similar parts according to the similarity between the sub-features comprises:

[0156] constructing a similarity matrix according to the similarity between all the sub-features of the text segments of the at least two images;

[0157] determining a minimum cost path of the similarity matrix, and taking the corresponding relationship of the sub-features represented by the minimum cost path as the corresponding relationship of the associated sub-features.

[0158] In an alternative embodiment, the similarity between the image features of the text segments in the pair of associated image samples is determined according to the correspondence between the sub-features after the association.

[0159] The similarity between the image features of the text segments in the pair of associated image samples is determined according to the similarity of the sub-features on the minimum cost path.

[0160] In an alternative embodiment, the pair of associated image samples is obtained by:

[0161] extracting the image frames of the text segments from the image frames of the video samples;

[0162] The image frames of the text segments with similar parts are taken as the pair of associated image samples.

[0163] In an alternative embodiment, the method further comprises:

[0164] The image frames of the text segments without similar parts are taken as the pair of non-associated image samples;

[0165] The similarity between the image features of the text segments in the pair of non-associated image samples is determined by similarity calculation on the similar parts of the text segments in the pair of non-associated image samples.

[0166] The image information parsing network is trained according to the similarity determined for the pair of associated image samples.

[0167] The image information parsing network is trained according to the similarity determined for the pair of associated image samples and the similarity determined for the pair of non-associated image samples.

[0168] The present application can also take the image frames of the text segments with similar parts as the pair of associated image samples, and take the image frames of the text segments without similar parts as the pair of non-associated image samples, that is, a ternary sample group including positive sample pairs and negative sample pairs can be constructed, and the loss function of the positive sample pairs and the loss function of the negative sample pairs are combined for training, so that an image information parsing network with more accurate representation can be obtained.

[0169] In an alternative embodiment, the similarity between the image features of the text segments in the at least two images is determined by similarity calculation on the similar parts of the text segments in the at least two images.

[0170] The similarity between the image features of the text segments in the at least two images is determined by similarity calculation on the similar parts of the text segments in the at least two images.

[0171] In an alternative embodiment, the determining the similarity between the image features of the text segments in the at least two images by performing similarity calculation on the similar parts of the text segments in the at least two images comprises:

[0172] dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of the at least two images;

[0173] associating the sub-features including the similar parts according to the similarity between the sub-features;

[0174] determining the similarity between the image features of the text segments in the at least two images according to the corresponding relationship of the sub-features after the association.

[0175] In an alternative embodiment, the associating the sub-features including the similar parts according to the similarity between the sub-features comprises:

[0176] constructing a similarity matrix according to the similarity between all the sub-features of the text segments of the at least two images;

[0177] determining a minimum cost path of the similarity matrix, and taking the corresponding relationship of the sub-features represented by the minimum cost path as the corresponding relationship of the sub-features after the association.

[0178] In an alternative embodiment, the determining the similarity between the image features of the text segments in the at least two images according to the corresponding relationship of the sub-features after the association comprises:

[0179] determining the similarity between the image features of the text segments in the at least two images according to the similarity of the sub-features on the minimum cost path.

[0180] In an alternative embodiment, the merging the text segments in the adjacent two image frames according to the similarity comprises:

[0181] performing deduplication processing and / or padding processing on the text segments in the at least two images with a similarity satisfying a set range.

[0182] In an alternative embodiment, the method further comprises:

[0183] providing the obtained text content on a device interface;

[0184] displaying updated text content based on an editing operation on the obtained text content.

[0185] Embodiments of training an image information parsing network are given as follows, with reference to Figure 3, a flowchart of a processing method of an image information parsing network is shown, and the method can specifically include the following steps:

[0186] In step 201, a pair of associated image samples is obtained.

[0187] In step 202, image features of text segments in the pair of associated image samples are extracted respectively.

[0188] In step 203, similarity between the image features of the text segments in the pair of associated image samples is determined by performing similarity calculation on similar parts of the text segments in the pair of associated image samples.

[0189] In step 204, the image information parsing network is trained according to the similarity determined for the pair of associated image samples, for extracting image features of text segments in images.

[0190] Compared with the prior scheme of extracting pre-designed image features, the image information parsing network for image feature extraction of text segments in the embodiments of the present application is trained according to the similarity between the image features of text segments in the pair of associated image samples, which can better adaptively learn image features under projection transformation, light shielding, jitter blur, etc., can be applicable to the case where the form and content of text segments change greatly in the video process, and can more accurately associate text segments of video frames, so that the extraction result of video content is simple and complete, meets the requirement of output content, and improves the user experience.

[0191] Since the image features are extracted based on the image information parsing network, the image information parsing network is trained according to the similarity between the image features of text segments in the pair of associated image samples, and the similarity between the image features is determined by performing similarity calculation on similar parts of the text segments in the pair of associated image samples, that is, the similar parts are aligned after dynamic matching of the text segments in the pair of associated image samples, and then the similarity calculation is performed. Compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, does not need to label the sequence characteristics between the text segments, can better adapt to the sequence characteristics of partially overlapping text segments between video frames in the video scene, has stronger universality, and the image features extracted by the image information parsing network trained according to the similarity are more robust in dealing with the sequence characteristics of text segments, and thus when used for similarity calculation of text segments, the text segments of adjacent video frames can be better associated.

[0192] The following gives an embodiment of training the image information parsing network according to a three-sample group, with reference to Figure 4, a flowchart of a processing method of an image information parsing network is shown, and the method can specifically include the following steps:

[0193] Step 301, obtaining a triple sample set including associated image sample pairs and non-associated image sample pairs;

[0194] Step 302, respectively extracting image features of the text segments in the triple sample set;

[0195] Step 303, determining the similarity between the image features of the text segments in the associated image sample pairs by performing similarity calculation on the similar parts of the text segments in the associated image sample pairs;

[0196] Step 304, determining the similarity between the image features of the text segments in the non-associated image sample pairs by performing similarity calculation on the similar parts of the text segments in the non-associated image sample pairs;

[0197] Step 305, training the image information parsing network according to the similarity determined for the associated image sample pairs and the similarity determined for the non-associated image sample pairs, for extracting image features of text segments in images.

[0198] Compared with the prior scheme of extracting pre-designed image features, the image information parsing network for text segment image feature extraction in the present application is trained according to the similarity between the image features of the text segments in the associated image sample pairs, which can better adaptively learn the image features under projection transformation, light shielding, shaking blur, etc., can be applicable to the case where the form and content of the text segments change greatly in the video process, and can more accurately associate the text segments of the video frames, so that the extraction result of the video content is simple and complete, meets the requirement of output content, and improves the user's use experience.

[0199] Since the image features are extracted based on the image information parsing network, the image information parsing network is trained according to the similarity between the image features of the Chinese text segments in the associated image sample pairs, and the similarity between the image features is determined by similarity calculation on the similar parts of the Chinese text segments in the associated image sample pairs, that is, the similar parts are matched and aligned, and then the similarity calculation is performed. Compared with the scheme of directly calculating the similarity of the text segments, the obtained similarity more accurately represents the similarity between the text segments, without the need for labeling the sequence characteristics between the text segments, and can better adapt to the sequence characteristics of the partially overlapping text segments in the video frames in the video scene, and has stronger universality. The image features extracted by the image information parsing network trained according to the similarity are more robust to the sequence characteristics of the text segments, and thus when used for similarity calculation of the text segments, the text segments of adjacent video frames can be better associated.

[0200] The application can also use the image frames of the text segments with similar parts as associated image sample pairs, and the image frames of the text segments without similar parts as non-associated image sample pairs, that is, a three-sample group including positive sample pairs and negative sample pairs can be constructed, and the loss function of the positive sample pairs and the loss function of the negative sample pairs are trained simultaneously, so that a more accurate image information parsing network can be obtained.

[0201] The image features of the text segments are used for image association retrieval as follows: a second image similar to a first image is calculated according to the image features extracted by the image information parsing network, and then the second image is provided as a retrieval result. Referring to Figure 5 , a flowchart of an image retrieval method according to an embodiment of the application is shown, which can specifically include the following steps:

[0202] Step 401, obtaining a first image for retrieval;

[0203] Step 402, extracting image features of text segments in the first image;

[0204] Step 403, determining the similarity between the image features of the text segments in the first image and the text segments in a second image by similarity calculation on the similar parts of the text segments included in the first image and the second image respectively;

[0205] Step 404, determining the second image as an associated image of the first image according to the similarity;

[0206] Step 405, providing the associated image as an image retrieval result.

[0207] According to the embodiment of the present application, after the image features of the text segments in the two images are extracted, the similarity between the image features of the text segments in the two images is determined by performing similarity calculation on the similar parts of the text segments in the two images, and the text segments are further merged according to the similarity, so as to obtain the text content corresponding to the two images. Since the similarity between the image features is determined by performing similarity calculation on the similar parts of the text segments in the two images, that is, the alignment processing is performed after the dynamic matching of the text segments in the two images, and the similarity calculation is performed after the corresponding of the similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, without the need for labeling the sequence characteristics between the text segments, the image text segments can be better associated, the extraction result of the video content is simple and complete, the requirement of the output content is met, and the use experience of the user is improved. Moreover, the sequence characteristics of the partially overlapped text segments between the video frames in the video scene can be better adapted, the universality is stronger, and when the similarity calculation of the text segments is performed, the similar images can be better searched.

[0208] The scheme of the present application for associating and processing the text segments in at least two images can be applied to the processing of text segments in educational live broadcast, entertainment live broadcast, or remote course video, news video, street view video, commodity video, and entertainment video. By associating the text segments of two adjacent image frames in the video, the text content corresponding to the video can be generated. For the text content obtained in the education scene, the directory structure and knowledge content in the video can be quickly converted into text information, which is convenient for users to record and quickly find; for the text segments obtained in the entertainment scene, they can be used as editing materials for subtitles or special effects; for the case that the subtitles or special effect text in the video are missing, the missing part of the content can be filled, and the text segments displayed by the video frame are further updated.

[0209] The following embodiment of applying video content extraction to the course video scene is given, with reference to Figure 6 , a flowchart of a course content extraction method according to the fifth embodiment of the present application is shown, which can specifically include the following steps:

[0210] Step 501, extracting image features of text segments in image frames of a course video;

[0211] Step 502, determining the similarity between the image features of the text segments corresponding to two adjacent image frames by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames;

[0212] Step 503, merging the text segments in the two adjacent image frames with the similarity satisfying the set range, to obtain the text content corresponding to the course video.

[0213] According to the embodiment of the present application, when the content of the course video is extracted, based on the relevance of the text segments displayed in the video, after the image features of the text segments in the image frames of the course video are extracted, the similarity between the image features of the text segments in the adjacent two image frames is determined, and the text segments are further merged according to the similarity, so as to obtain the text content corresponding to the course video, that is, the course content in the form of text, which is convenient for subsequent learning and use.

[0214] According to the embodiment of the present application, after the image features of the text segments in the adjacent two image frames are extracted, the similarity between the image features of the text segments in the adjacent two image frames is determined by similarity calculation on the similar parts of the text segments in the adjacent two image frames, and the text segments are further merged according to the similarity, so as to obtain the text content corresponding to the adjacent two image frames. Since the similarity between the image features is determined by similarity calculation on the similar parts of the text segments in the adjacent two image frames, that is, the alignment processing is performed after the dynamic matching of the text segments in the adjacent two image frames, and the similarity calculation is performed after the corresponding of the similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, without labeling the sequence characteristics between the text segments, the image text segments can be better associated, the extraction result of the video content is simple and complete, the requirement of the output content is met, and the use experience of the user is improved. Moreover, it can better adapt to the sequence characteristics of the partially overlapped text segments between the video frames in the video scene, and has higher universality.

[0215] The following embodiment shows the application of the video content extraction to the video subtitle extraction, with reference to Figure 7 , a flowchart of a subtitle extraction method according to the embodiment six of the present application is shown, which can specifically include the following steps:

[0216] Step 601, image features of the text segments are respectively identified from the image frames of the video;

[0217] Step 602, similarity between the image features of the text segments corresponding to the adjacent two image frames is determined by similarity calculation on the similar parts of the text segments in the adjacent two image frames;

[0218] Step 603, the text segments in the adjacent two image frames with the similarity satisfying the set range are merged, and the subtitle content corresponding to the video is obtained.

[0219] In an optional embodiment, the similarity between the image features of the text segments corresponding to the adjacent two image frames is determined by similarity calculation on the similar parts of the text segments in the adjacent two image frames, which includes:

[0220] The image features are divided into a plurality of sub-features in the feature space width, and the similarity between the sub-features of the text segments of the two adjacent image frames is determined;

[0221] According to the similarity between the sub-features, the sub-features including the similar parts are associated;

[0222] According to the corresponding relationship of the associated sub-features, the similarity between the image features of the text segments in the two adjacent image frames is determined.

[0223] In an optional embodiment, the subtitle segments in the two adjacent image frames with the similarity satisfying the set range are merged.

[0224] The text segments in the two adjacent image frames with the similarity satisfying the set range are de-duplicated and / or supplemented.

[0225] According to the embodiments of the present application, after extracting the image features of the subtitle segments in the two adjacent image frames, the similarity between the image features of the subtitle segments in the two adjacent image frames is determined by performing similarity calculation on the similar parts of the subtitle segments in the two adjacent image frames, and the subtitle segments are further merged according to the similarity, so as to obtain the text content corresponding to the two adjacent image frames. Since the similarity between the image features is determined by performing similarity calculation on the similar parts of the subtitle segments in the two adjacent image frames, that is, the alignment processing is performed after dynamically matching the subtitle segments in the two adjacent image frames, and the similarity calculation is performed after corresponding the similar parts, compared with the scheme of directly performing similarity calculation on the subtitle segments, the obtained similarity more accurately represents the similarity between the subtitle segments, without labeling the sequence characteristics between the subtitle segments, the image subtitle segments can be better associated, so that the extraction result of the video content is simple and complete, the requirement of the output content is met, and the use experience of the user is improved. Moreover, the sequence characteristics of the partially overlapped subtitle segments between the video frames in the video scene can be better adapted, and the universality is stronger.

[0226] The dynamic matching and alignment of the text segments are performed to improve the accuracy of the similarity calculation, and the text segments of the video frames are better associated, as follows: Figure 8 The flowchart of a video content extraction method according to an embodiment of the present application is shown, and the method can specifically include the following steps:

[0227] Step 701, extracting the image features of the text segments in the image frames of the target video;

[0228] Step 702, determining the similarity between the image features of the text segments in the two adjacent image frames by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames;

[0229] Step 703, merging the text segments in the two adjacent image frames according to the similarity, to obtain the text content corresponding to the target video.

[0230] According to the embodiment of the present application, after extracting the image features of the text segments in the two adjacent image frames in the video, the similarity between the image features of the text segments in the two adjacent image frames is determined by similarity calculation on the similar parts of the text segments in the two adjacent image frames, and further, the text segments are merged according to the similarity, so as to obtain the text content corresponding to the two adjacent image frames. Since the similarity between the image features is determined by similarity calculation on the similar parts of the text segments in the two adjacent image frames, that is, the alignment processing is performed after dynamic matching of the text segments in the two adjacent image frames, and the similarity calculation is performed after corresponding the similar parts, compared with the scheme of directly performing similarity calculation on the text segments, the obtained similarity more accurately represents the similarity between the text segments, without labeling the sequence characteristics between the text segments, the image text segments can be better associated, so that the extraction result of the video content is simple and complete, the requirement of the output content is met, and the user's use experience is improved. Moreover, it can better adapt to the sequence characteristics of the partially overlapping text segments between the video frames in the video scene, and has stronger universality.

[0231] The following embodiment of applying video content extraction to text segment processing in a video conference scene is given, with reference to Figure 9 , a flowchart of a conference content processing method according to the eighth embodiment of the present application is shown, which can specifically include the following steps:

[0232] Step 801, acquiring a conference video in real time;

[0233] Step 802, extracting image features of text segments in image frames of the conference video;

[0234] Step 803, determining the similarity between the image features of the text segments corresponding to the two adjacent image frames respectively by similarity calculation on the similar parts of the text segments in the two adjacent image frames;

[0235] Step 804, merging the text segments in the two adjacent image frames whose similarity meets the set range;

[0236] Step 805, adding the text content obtained after merging as subtitles to the conference video;

[0237] Step 806, providing the conference video with subtitles.

[0238] The remote conference video can display courseware, files, PPT content display, which includes key information related to the conference content, so that the text segments in the video can be extracted, and the text segments can be merged as subtitles based on the relevance between the text segments in the image frames. Compared with the display in the image frames, the subtitles are more concise and clear, and the key information in the conference video can be obtained in time, so that the user participating in the video can view the subtitles during the video conference or after the conference.

[0239] The conference video can be obtained in real time in the video conference, and the image features of the text segments in the image frames are further extracted. The similarity between the image features of the text segments corresponding to two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames. If the similarity meets the set condition, it can be considered that the text segments of the adjacent image frames are associated, and the text segments can be further merged, such as de-duplication of the same text content, or using the associated text segments to fill in the incomplete display part. The merged text content is added as subtitles to the conference video, and the conference video with subtitles is provided to the conference client for user viewing.

[0240] The following embodiments of applying video content extraction to a remote video scenario are given with reference to Figure 10 , a flowchart of a remote video processing method according to Embodiment Nine of the present application is shown, which can specifically include the following steps:

[0241] Step 901, identifying the image features of the text segments in the image frames of the remote medical video;

[0242] Step 902, determining the similarity between the image features of the text segments corresponding to two adjacent image frames by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames;

[0243] Step 903, merging the text segments in the two adjacent image frames with a similarity meeting the set range;

[0244] Step 904, updating the text segments in the remote medical video frames according to the text content obtained after the merging;

[0245] Step 905, providing the updated remote medical video.

[0246] Remote medical treatment can be used to solve many medical problems that cannot be handled in close proximity. The video may involve the display of the surface text of a packaging box of a file, medicine or medical device, and the text content is closely related to the remote medical treatment. Since the display may not be complete or lack, it is important to extract and process the text fragments according to the association of multiple video frames. Compared with the display in the image frame, the text fragments are more concise and clear, which is convenient for obtaining key medical information in time for users participating in the remote medical video to view during the video or after the video.

[0247] The image features of the text fragments in the image frames in the remote medical video can be extracted. The similarity between the image features of the text fragments corresponding to two adjacent image frames is determined by calculating the similarity of the similar parts of the text fragments in the two adjacent image frames. If the similarity meets the set condition, it can be considered that the text fragments of the adjacent image frames are associated, and the text fragments can be further merged, such as de-duplication of the same text content, or using the associated text fragments to fill in the incomplete display part. The text content obtained after the merging processing can be used to update the text fragments in the video, and the updated video is provided to the terminal of the remote medical video for real-time viewing or post-video viewing.

[0248] The following embodiments give the cloud interaction process in the text content extraction process in the image. Referring to Figure 11 , a flowchart of a text content extraction method according to Embodiment Ten of the present application is shown, which can specifically include the following steps:

[0249] Step 1001, obtaining at least two images submitted;

[0250] Step 1002, extracting image features of text fragments in the at least two images;

[0251] Step 1003, determining the similarity between the image features of the text fragments in the at least two images by calculating the similarity of the similar parts of the text fragments in the at least two images;

[0252] Step 1004, merging the text fragments in the at least two images according to the similarity to obtain the text content corresponding to the at least two images;

[0253] Step 1005, providing the text content.

[0254] A terminal user such as a video platform (a video software service provider), a subtitle / video / image editing professional, or a general user can access a cloud service platform through a set video interface and submit at least two images requiring extraction of text content. The cloud service platform provides a corresponding service for a video or an image by using its own computing resources, extracts image features of text segments in the submitted images, determines the similarity between the image features of the text segments in the images by performing similarity calculation on similar parts of the text segments in the images, further merges the text segments in the images according to the similarity, and obtains corresponding text content. Figure 22 FIG. 10 shows an interaction schematic diagram of a terminal and a cloud service platform. After the terminal submits at least two images, the cloud service platform feeds back corresponding text content to the terminal for viewing by a terminal user or personalized editing as required.

[0255] The interface parameters can define a video sampling rate (for example, sampling once every 5 seconds, sampling one image frame each time), a resolution (the image can be enlarged or reduced according to processing efficiency and clarity requirements), the cloud service platform can periodically generate logs to record data such as processing time and resource occupation of each processing, and can also statistically process video processing of different video platforms or terminals to facilitate subsequent optimization of the cloud service. The feedback content can include extracted text content, position coordinates of the text segments in the images, start or end times of the text segments, and can be segmented and displayed or separated by adding interval symbols for differentiation in the case of multiple text segments in one video frame.

[0256] Referring to Figure 12 FIG. 11 shows a structural block diagram of a text content extraction device according to Embodiment Eleven of the present application, which can specifically include:

[0257] The feature extraction module 1101 is configured to extract image features of text segments in at least two images.

[0258] The similarity determination module 1102 is configured to determine the similarity between the image features of the text segments in the at least two images by performing similarity calculation on similar parts of the text segments in the at least two images.

[0259] The merging module 1103 is configured to merge the text segments in the at least two images according to the similarity to obtain corresponding text content of the at least two images.

[0260] In an optional embodiment, the feature extraction module is specifically configured to divide an image region where the text segment is located into a plurality of character regions, extract image features of the character regions respectively, and merge the image features of the character regions into the image features of the text segment.

[0261] In an optional embodiment, the device further includes:

[0262] a network training module configured to obtain a pair of associated image samples and train an image information analysis network for extracting image features of text segments in images according to the pair of associated image samples.

[0263] In an optional embodiment, the network training module comprises:

[0264] a feature extraction sub-module configured to extract image features of text segments in the pair of associated image samples respectively;

[0265] a similarity determination sub-module configured to determine similarities between the image features of the text segments in the pair of associated image samples by performing similarity calculation on similar parts of the text segments in the pair of associated image samples;

[0266] a similarity training sub-module configured to train the image information analysis network for extracting image features of text segments in images according to the similarities determined for the pair of associated image samples.

[0267] In an optional embodiment, the similarity determination sub-module comprises:

[0268] a sub-feature division sub-unit configured to divide the image features into a plurality of sub-features in a feature space width;

[0269] a similarity determination sub-unit configured to determine similarities between the sub-features of the text segments of the at least two images;

[0270] a sub-feature association sub-unit configured to associate the sub-features including the similar parts according to the similarities between the sub-features;

[0271] a similarity determination sub-unit configured to determine the similarities between the image features of the text segments in the at least two images according to the corresponding relationship of the associated sub-features.

[0272] In an optional embodiment, the sub-feature association sub-unit is specifically configured to construct a similarity matrix according to the similarities between all the sub-features of the text segments of the at least two images, determine a minimum cost path of the similarity matrix, and take the corresponding relationship of the sub-features represented by the minimum cost path as the corresponding relationship of the associated sub-features.

[0273] In an optional embodiment, the merging module is specifically configured to perform deduplication processing and / or padding processing on the text segments in the at least two images with similarities satisfying a set range.

[0274] In an optional embodiment, the apparatus further comprises:

[0275] a content providing module configured to provide the obtained text content on a device interface.

[0276] a content updating module, configured to display updated literal content based on an editing operation on the obtained literal content.

[0277] With reference to Figure 13 , a structural block diagram of a processing apparatus of an image information parsing network according to an embodiment of the present application is shown, and specifically can include:

[0278] An image pair obtaining module 1201 is configured to obtain a pair of associated image samples.

[0279] An image feature extracting module 1202 is further configured to extract image features of literal segments in the pair of associated image samples, respectively.

[0280] A similarity calculating module 1203 is configured to determine similarities between the image features of the literal segments in the pair of associated image samples by performing similarity calculation on similar parts of the literal segments in the pair of associated image samples.

[0281] A network training module 1204 is configured to train the image information parsing network according to the determined similarities for the pair of associated image samples, for extracting image features of literal segments in images.

[0282] With reference to Figure 14 , a structural block diagram of a processing apparatus of an image information parsing network according to an embodiment of the present application is shown, and specifically can include:

[0283] A sample group obtaining module 1301 is configured to obtain a triple sample group including a pair of associated image samples and a pair of non-associated image samples.

[0284] An image feature extracting module 1302 is configured to extract image features of literal segments in the triple sample group, respectively.

[0285] A similarity calculating module 1303 is configured to determine similarities between the image features of the literal segments in the pair of associated image samples by performing similarity calculation on similar parts of the literal segments in the pair of associated image samples.

[0286] A similarity calculating module 1304 is configured to determine similarities between the image features of the literal segments in the pair of non-associated image samples by performing similarity calculation on similar parts of the literal segments in the pair of non-associated image samples.

[0287] A network training module 1305 is configured to train the image information parsing network according to the determined similarities for the pair of associated image samples and the determined similarities for the pair of non-associated image samples, for extracting image features of literal segments in images.

[0288] With reference to Figure 15, and shows a structural block diagram of an image retrieval device according to an embodiment fourteen of the present application, which can specifically include:

[0289] The first image acquisition module 1401 is configured to acquire a first image for retrieval;

[0290] The feature extraction module 1402 is configured to extract image features of a text segment in the first image;

[0291] The similarity calculation module 1403 is configured to determine a similarity between the image features of the text segment in the first image and a text segment in a second image by performing similarity calculation on similar parts of the text segments respectively included in the first image and the second image;

[0292] The image association module 1404 is configured to determine the second image as an associated image of the first image according to the similarity;

[0293] The result providing module 1405 is configured to provide the associated image as an image retrieval result.

[0294] Referring to Figure 16 , and shows a structural block diagram of a course content extraction device according to an embodiment fifteen of the present application, which can specifically include:

[0295] The feature extraction module 1501 is configured to extract image features of a text segment in an image frame of a course video;

[0296] The similarity calculation module 1502 is configured to determine a similarity between the image features of the text segments respectively corresponding to two adjacent image frames by performing similarity calculation on similar parts of the text segments in the two adjacent image frames;

[0297] The merging module 1503 is configured to perform merging processing on the text segments in the two adjacent image frames with the similarity satisfying a set range, to obtain text content corresponding to the course video.

[0298] Referring to Figure 17 , and shows a structural block diagram of a subtitle extraction device according to an embodiment sixteen of the present application, which can specifically include:

[0299] The feature recognition module 1601 is configured to recognize image features of a subtitle segment from image frames of a target video respectively;

[0300] The similarity calculation module 1602 is configured to determine a similarity between the image features of the subtitle segments respectively corresponding to two adjacent image frames by performing similarity calculation on similar parts of the text segments in the two adjacent image frames;

[0301] The merging module 1603 is configured to merge the text segments in the two adjacent image frames that satisfy the set range of similarity, to obtain the text content corresponding to the target video.

[0302] In an optional embodiment, the similarity calculation module is specifically configured to divide the image features into a plurality of sub-features in the feature space width, determine the similarity between the sub-features of the text segments of the two adjacent image frames, associate the sub-features including the similar parts according to the similarity between the sub-features, and determine the similarity between the image features of the text segments in the two adjacent image frames according to the corresponding relationship of the associated sub-features.

[0303] In an optional embodiment, the merging module is specifically configured to perform deduplication processing and / or padding processing on the text segments in the two adjacent image frames that satisfy the set range of similarity.

[0304] Referring to Figure 18 , a structural block diagram of a video content extraction device according to Embodiment Seventeen of the present application is shown, which can specifically include:

[0305] The feature extraction module 1701 is configured to extract the image features of the text segments in the image frames of the target video.

[0306] The similarity calculation module 1702 is configured to determine the similarity between the image features of the text segments in the two adjacent image frames by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames.

[0307] The merging module 1703 is configured to perform merging processing on the text segments in the two adjacent image frames according to the similarity, to obtain the text content corresponding to the target video.

[0308] Referring to Figure 19 , a structural block diagram of a conference content processing device according to Embodiment Eighteen of the present application is shown, which can specifically include:

[0309] The video conference acquisition module 1801 is configured to acquire a conference video in real time.

[0310] The feature extraction module 1802 is configured to extract the image features of the text segments in the image frames of the conference video.

[0311] The similarity calculation module 1803 is configured to determine the similarity between the image features of the text segments in the two adjacent image frames by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames.

[0312] The merging module 1804 is configured to merge the text segments in the two adjacent image frames that satisfy the set range of similarity.

[0313] The subtitle adding module 1805 is configured to add the text content obtained after the merging processing as a subtitle to the conference video.

[0314] The video providing module 1806 is configured to provide the conference video after adding the subtitle.

[0315] With reference to Figure 20 , a structural block diagram of a remote video processing apparatus according to an embodiment of the present application is shown, which can specifically include:

[0316] The feature extraction module 1901 is configured to identify image features of text segments in image frames of the remote medical video.

[0317] The similarity calculation module 1902 is configured to determine a similarity between the image features of the text segments corresponding to two adjacent image frames by performing similarity calculation on similar parts of the text segments in the two adjacent image frames.

[0318] The merging module 1903 is configured to perform merging processing on the text segments in the two adjacent image frames with a similarity meeting a set range.

[0319] The text updating module 1904 is configured to update the text segments in the remote medical video frames according to text content obtained after the merging processing.

[0320] The video providing module 1905 is configured to provide the updated remote medical video.

[0321] With reference to Figure 21 , a structural block diagram of a text content extraction apparatus according to an embodiment of the present application is shown, which can specifically include:

[0322] The image acquisition module 2001 is configured to acquire at least two images submitted.

[0323] The feature extraction module 2002 is configured to extract image features of text segments in the at least two images.

[0324] The similarity calculation module 2003 is configured to determine a similarity between the image features of the text segments in the at least two images by performing similarity calculation on similar parts of the text segments in the at least two images.

[0325] The merging module 2004 is configured to perform merging processing on the text segments in the at least two images according to the similarity, to obtain text content corresponding to the at least two images.

[0326] The content providing module 2005 is configured to provide the text content.

[0327] For the apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the description of the method embodiments.

[0328] Embodiments of the present disclosure can be implemented as a system configured to perform desired operations using any suitable hardware, firmware, software, or any combination thereof. Figure 22 An exemplary system (or apparatus) 2100 that can be used to implement various embodiments described in the present disclosure is shown schematically.

[0329] For one embodiment, Figure 21 An exemplary system 2100 is shown having one or more processors 2102, a system control module (chipset) 2104 coupled to at least one of the processors 2102, system memory 2106 coupled to the system control module 2104, non-volatile memory (NVM) / storage 2108 coupled to the system control module 2104, one or more input / output devices 2110 coupled to the system control module 2104, and a network interface 2112 coupled to the system control module 2104.

[0330] The processor(s) 2102 can include one or more single core or multi core processors, which can include general purpose processors or special purpose processors (e.g., graphics processors, application processors, baseband processors, etc.) in any combination. In some embodiments, the system 2100 can be capable of acting as a browser as described in embodiments of the present disclosure.

[0331] In some embodiments, the system 2100 can include one or more computer readable media (e.g., system memory 2106 or NVM / storage 2108) having instructions and one or more processors 2102 incorporated with the one or more computer readable media configured to execute the instructions to implement modules to perform the actions described in the present disclosure.

[0332] For one embodiment, the system control module 2104 can include any suitable interface controllers to provide for any suitable interface to at least one of the processor(s) 2102 and / or any suitable device or component in communication with the system control module 2104.

[0333] The system control module 2104 can include a memory controller module to provide an interface to the system memory 2106. The memory controller module can be a hardware module, a software module, and / or a firmware module.

[0334] System memory 2106 can be used to load and store data and / or instructions, for example, for system 2100. For one embodiment, system memory 2106 can include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 2106 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0335] For one embodiment, system control module 2104 can include one or more input / output controllers to provide an interface to NVM / storage device 2108 and input / output device(s) 2110.

[0336] For example, NVM / storage device 2108 can be used to store data and / or instructions. NVM / storage device 2108 can include any suitable non-volatile memory (e.g., flash memory) and / or can include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0337] NVM / storage device 2108 can include a storage resource that is physically part of system 2100 on which the device is installed, or it can be accessible by the device over a network, without being physically part of the device. For example, NVM / storage device 2108 can be accessed by the device through input / output device(s) 2110 over a network.

[0338] Input / output device(s) 2110 can provide an interface between system 2100 and any suitable peripheral devices, including communication components, audio components, sensor components, and the like. Network interface 2112 can provide an interface for system 2100 to communicate over one or more networks, and system 2100 can wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as to access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof.

[0339] For one embodiment, at least one of the processor(s) 2102 can be packaged together with logic for one or more controllers of the system control module 2104 (e.g., a memory controller module). For one embodiment, at least one of the processor(s) 2102 can be packaged together with logic for one or more controllers of the system control module 2104 to form a system in a package (SiP). For one embodiment, at least one of the processor(s) 2102 can be fabricated together with logic for one or more controllers of the system control module 2104 on the same die. For one embodiment, at least one of the processor(s) 2102 can be fabricated together with logic for one or more controllers of the system control module 2104 on the same die to form a system on a chip (SoC).

[0340] In various embodiments, the system 2100 can be, but is not limited to, a browser, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, the system 2100 can have more or less components, and / or different architectures. For example, in some embodiments, the system 2100 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including touch screen displays), non- volatile memory ports, multiple antennas, graphics chips, application specific integrated circuits (ASICs), and speakers.

[0341] In various embodiments, the display includes a touch panel, the display screen can be implemented as a touch screen display to receive input signals from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect duration and pressure related to the touch or slide action.

[0342] The embodiments of the present application further provide a non-transitory readable storage medium, which stores one or more programs, and the one or more programs, when applied in a terminal device, can enable the terminal device to execute instructions of the method steps in the embodiments of the present application.

[0343] In one example, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method in the embodiments of the present application when executing the computer program.

[0344] In one example, a computer readable storage medium is also provided, which stores a computer program, and the program, when executed by a processor, implements the method of one or more of the embodiments of the present application.

[0345] While certain embodiments have been illustrated and described, it will be understood by those skilled in the art that various changes in form and / or substitutions of equivalents can be made without departing from the spirit and scope of the application. The application is intended to cover any and all such modifications or variations. It is therefore evident that the embodiments described herein have a wide range of applications. It should be apparent that the application is not limited to the embodiments described above, but can be practiced with modification and change by one having ordinary skill in the art without departing from the scope of the application.

Claims

1. A method of extracting literal content, characterized by, The method comprises: extracting image features of text segments in at least two images; determining the similarity between the image features of the text segments in the at least two images by performing similarity calculation on similar parts of the text segments in the at least two images, wherein the method comprises: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments in the at least two images; associating the sub-features including the similar parts according to the similarity between the sub-features; and determining the similarity between the image features of the text segments in the at least two images according to the corresponding relationship of the associated sub-features; performing merging processing on the text segments in the at least two images according to the similarity, to obtain the corresponding text content of the at least two images.

2. The method of claim 1, wherein, The method comprises: dividing the image area where the text segment is located into a plurality of character areas; extracting image features of the character areas respectively, and merging the image features of the character areas into the image features of the text segment.

3. The method of claim 1, wherein, The method further comprises: obtaining a pair of associated image samples, and training an image information analysis network for extracting image features of text segments in images according to the pair of associated image samples.

4. The method of claim 3, wherein, The method comprises: extracting image features of text segments in the pair of associated image samples respectively; determining the similarity between the image features of the text segments in the pair of associated image samples by performing similarity calculation on similar parts of the text segments in the pair of associated image samples; training the image information analysis network for extracting image features of text segments in images according to the similarity determined for the pair of associated image samples.

5. The method of claim 1, wherein, The method comprises: constructing a similarity matrix according to the similarity between all sub-features of the text segments in the at least two images; determining the minimum cost path of the similarity matrix, and taking the corresponding relationship of the sub-features represented by the minimum cost path as the corresponding relationship of the associated sub-features.

6. The method of claim 1, wherein, The method comprises: performing deduplication processing and / or padding processing on the text segments in the at least two images whose similarity satisfies a set range.

7. The method of claim 1, wherein, The method further comprises: providing the obtained text content on a device interface; displaying updated text content based on an editing operation on the obtained text content.

8. A processing method of an image information analysis network, characterized by, The method comprises: obtaining a pair of associated image samples; extracting image features of text segments in the pair of associated image samples respectively; Determine the similarity between the image features of the text segments in the associated image sample pair by performing similarity calculation on the similar parts of the text segments in the associated image sample pair, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of at least two images; according to the similarity between the sub-features, associating the sub-features including the similar parts; according to the corresponding relationship of the associated sub-features, determine the similarity between the image features of the text segments in the at least two images; Train the image information analysis network according to the similarity determined for the associated image sample pair, for extracting the image features of the text segments in the image.

9. A processing method of an image information analysis network, characterized by, Including: Obtain a triple sample group including an associated image sample pair and a non-associated image sample pair; Respectively extract the image features of the text segments in the triple sample group; Determine the similarity between the image features of the text segments in the associated image sample pair by performing similarity calculation on the similar parts of the text segments in the associated image sample pair, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of at least two images; according to the similarity between the sub-features, associating the sub-features including the similar parts; according to the corresponding relationship of the associated sub-features, determine the similarity between the image features of the text segments in the at least two images; Determine the similarity between the image features of the text segments in the non-associated image sample pair by performing similarity calculation on the similar parts of the text segments in the non-associated image sample pair; Train the image information analysis network according to the similarity determined for the associated image sample pair and the similarity determined for the non-associated image sample pair, for extracting the image features of the text segments in the image.

10. An image retrieval method characterized by, Including: Obtain a first image for retrieval; Extract the image features of the text segments in the first image; Determine the similarity between the image features of the text segments in the first image and the second image by performing similarity calculation on the similar parts of the text segments respectively included in the first image and the second image, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of at least two images; according to the similarity between the sub-features, associating the sub-features including the similar parts; according to the corresponding relationship of the associated sub-features, determine the similarity between the image features of the text segments in the at least two images; According to the similarity, determine that the second image is the associated image of the first image; Provide the associated image as the image retrieval result.

11. A method of extracting course content, characterized by, Including: Extract the image features of the text segments in the image frames of the course video; The similarity between the image features of the text segments corresponding to two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of at least two images; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the associated sub-features, the similarity between the image features of the text segments in the at least two images is determined; The text segments in the two adjacent image frames with a similarity satisfying a set range are merged to obtain the text content corresponding to the course video.

12. A subtitle extraction method characterized by comprising: It includes: The image features of the text segments in the image frames of the target video are identified respectively; The similarity between the image features of the text segments corresponding to two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of the two adjacent image frames; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the associated sub-features, the similarity between the image features of the text segments in the at least two images is determined; The text segments in the two adjacent image frames with a similarity satisfying a set range are merged to obtain the text content corresponding to the course video.

13. The method of claim 12, wherein, The text segments in the two adjacent image frames with a similarity satisfying a set range are merged to obtain the text content corresponding to the course video. The text segments in the two adjacent image frames with a similarity satisfying a set range are merged to obtain the text content corresponding to the course video.

14. A method of video content extraction, characterized by, It includes: The image features of the text segments in the image frames of the target video are identified respectively; The similarity between the image features of the text segments corresponding to two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of at least two images; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the associated sub-features, the similarity between the image features of the text segments in the at least two images is determined; According to the similarity, the text segments in the two adjacent image frames are merged to obtain the text content corresponding to the target video.

15. A conference content processing method characterized by, It includes: Real-time acquisition of conference video; The image features of the text segments in the image frames of the conference video are extracted; The image features of the text segments in the image frames of the conference video are extracted; The similarity between the image features of the text segments corresponding to the two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of the at least two images; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the sub-features after association, the similarity between the image features of the text segments in the at least two images is determined; The text segments in the two adjacent image frames whose similarity meets the set range are merged; The text content obtained after merging is added to the conference video as subtitles. The conference video with subtitles is provided.

16. A method of remote video processing, the method comprising: It includes: Identifying the image features of the text segments in the image frames of the remote medical video; The similarity between the image features of the text segments corresponding to the two adjacent image frames is determined by performing similarity calculation on the similar parts of the text segments in the two adjacent image frames, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of the at least two images; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the sub-features after association, the similarity between the image features of the text segments in the at least two images is determined; The text segments in the two adjacent image frames whose similarity meets the set range are merged; The text content obtained after merging is added to the conference video as subtitles. The conference video with subtitles is provided.

17. A method of extracting literal content, characterized by, It includes: Obtaining at least two images submitted; Extracting the image features of the text segments in the at least two images; The similarity between the image features of the text segments in the at least two images is determined by performing similarity calculation on the similar parts of the text segments in the at least two images, including: dividing the image features into a plurality of sub-features in the feature space width, determining the similarity between the sub-features of the text segments of the at least two images; according to the similarity between the sub-features, the sub-features including the similar parts are associated; according to the corresponding relationship of the sub-features after association, the similarity between the image features of the text segments in the at least two images is determined; The text segments in the at least two images are merged according to the similarity, and the text content corresponding to the at least two images is obtained; The text content is provided.

18. An electronic device, comprising: It includes: A processor; And A memory having executable code stored thereon, which, when executed, causes the processor to perform the method of any one of claims 1-16.

19. One or more machine-readable media having stored thereon executable code that, when executed, causes a processor to perform the method of any one of claims 1-16.

Images