Text information extraction method and device, equipment and medium

Abstract

The invention provides a text information extraction method and device, equipment and a medium, and the method comprises the steps: obtaining a video stream containing a picture displayed by vehicle-mounted equipment; wherein the video stream is obtained by collecting a displayed picture in a vehicle testing process; performing text extraction on each video frame in the video stream to obtain initial text information; wherein the initial text information comprises all text contents in each video frame and an initial text area of all the text contents; and extracting interested text information from the initial text information of each video frame, and extracting target text content from the interested text information.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text processing, and in particular to a text information extraction method, device, equipment and medium. Background Art

[0002] With the continuous development of the automotive industry, the content, layout and style of car computers are becoming more diverse and personalized. At the same time, in the field of automotive testing, it is often necessary to obtain and parse the text information displayed by the car computer. In related technologies, testers usually record manually, take photos, or perform OCR (Optical Character Recognition) analysis on the car computer image at a specific moment during the test.

[0003] However, all of the above technical solutions have certain drawbacks. For example, on the one hand, facing a variety of car machine styles, the operation is complicated and inefficient; on the other hand, it is impossible to continuously parse the car machine content during the test process. Therefore, the relevant technical solutions are usually difficult to obtain the required text information efficiently and directly. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, apparatus, device and medium for extracting text information.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for extracting text information, comprising:

[0006] Acquire a video stream containing a screen displayed by the vehicle-mounted device; wherein the video stream is obtained by collecting the displayed screen during the vehicle testing process;

[0007] Performing text extraction on each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​the all text content;

[0008] The text information of interest is extracted from the initial text information of each of the video frames, and the target text content is extracted from the text information of interest.

[0009] In an optional implementation manner, extracting the text information of interest from the initial text information of each of the video frames includes:

[0010] In the case where the video frame is an initial frame in the video stream, the initial text information of the initial frame is subjected to text classification according to the purpose of text extraction through a text classification model to obtain the text information of interest of the initial frame.

[0011] In an optional implementation manner, extracting the text information of interest from the initial text information of each of the video frames includes:

[0012] When the video frame is a non-initial frame in the video stream, calculating an intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame;

[0013] When the intersection-and-union ratio meets the threshold requirement, the text information of interest of the video frame is determined according to the text region of interest corresponding to the maximum intersection-and-union ratio and the text content of interest corresponding to the text region of interest.

[0014] In an optional implementation manner, the calculating the intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame includes:

[0015] Performing region enlargement processing on each text region of interest in the previous video frame;

[0016] The intersection-over-union ratio between each of the initial text regions and the enlarged text region of interest is calculated.

[0017] In an optional implementation manner, the method further includes:

[0018] When the intersection-over-union ratio does not meet the threshold requirement, the initial text information of the non-initial frame is classified according to the purpose of text extraction by using a text classification model to obtain the text information of interest of the non-initial frame.

[0019] In an optional implementation manner, the calculating the intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame includes:

[0020] Determining affine transformation parameters between the video frame and the previous video frame;

[0021] The video frame and the previous video frame are image registered according to the affine transformation parameters, and the intersection-over-union ratio between the initial text region and each text region of interest in the previous video frame after registration is calculated.

[0022] In an optional implementation manner, determining the affine transformation parameters between the video frame and the previous video frame includes:

[0023] Determining a plurality of strong corner points in the previous video frame;

[0024] Determine the mapping position of each of the strong corner points in the video frame based on the optical flow method;

[0025] The affine transformation parameters are determined according to the positions of the multiple strong corner points in the previous video frame and the mapping positions.

[0026] In a second aspect, an embodiment of the present disclosure provides a text information extraction device, including:

[0027] An acquisition unit, used to acquire a video stream containing a picture displayed by the vehicle-mounted device; wherein the video stream is acquired by collecting the displayed picture during the vehicle testing process;

[0028] A first text extraction unit, configured to extract text from each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​the all text content;

[0029] The second text extraction unit is used to extract text information of interest from the initial text information of each of the video frames, and to extract target text content from the text information of interest.

[0030] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0031] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0032] The present disclosure provides a method, device, equipment and medium for extracting text information. In an embodiment of the present disclosure, during a vehicle test, a video stream containing a screen displayed by an on-board device can be collected; then, text extraction is performed on each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​all text content; interesting text information is extracted from the initial text information of each video frame, and target text content is extracted from the interesting text information.

[0033] In the above-mentioned embodiment, during the vehicle testing process, by acquiring the video stream of the screen displayed by the vehicle-mounted device and extracting all the text information in each video frame in the video stream, the target text content is extracted from all the text information. It is possible to extract the desired text content for vehicle-mounted devices of different styles; at the same time, the text content displayed by the vehicle-mounted device can be continuously acquired, so that the required text content can be acquired more efficiently, comprehensively and directly.

[0034] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of a text information extraction method provided by an embodiment of the present disclosure is shown;

[0037] Figure 2 A flowchart of another text information extraction method provided by an embodiment of the present disclosure is shown;

[0038] Figure 3 A schematic diagram of a text information extraction device provided by an embodiment of the present disclosure is shown;

[0039] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0041] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0042] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0043] With the continuous development of the automotive industry, the content, layout and style of vehicle-mounted equipment are becoming more diverse and personalized. In the field of automotive testing, it is often necessary to obtain and parse the text information displayed on the display interface of the vehicle-mounted equipment. In related technologies, the relevant text information can be obtained and parsed by the tester manually recording or taking photos; in addition, OCR analysis can be performed on the vehicle image at a specific moment during the test process to parse the corresponding text information.

[0044] However, with the rapid development of technology, the styles of vehicle-mounted devices are more diverse. At this time, the operation process of extracting text information described above is more complicated and less efficient. At the same time, the method described above cannot continuously parse the text content displayed by the vehicle-mounted device during the vehicle test. Therefore, the technical solution described above is difficult to efficiently and directly obtain the required text information.

[0045] Based on the above research, the present disclosure provides a text information extraction method, device, equipment and medium. In the embodiment of the present disclosure, during the vehicle test process, a video stream containing the screen displayed by the vehicle-mounted device can be collected; then, text extraction is performed on each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and the initial text area of ​​all text content; the text information of interest is extracted from the initial text information of each video frame, and the target text content is extracted from the text information of interest.

[0046] In the above-mentioned embodiment, during the vehicle testing process, by acquiring the video stream of the screen displayed by the vehicle-mounted device and extracting all the text information in each video frame in the video stream, the target text content is extracted from all the text information. It is possible to extract the desired text content for vehicle-mounted devices of different styles; at the same time, the text content displayed by the vehicle-mounted device can be continuously acquired, so that the required text content can be acquired more efficiently, comprehensively and directly.

[0047] To facilitate understanding of this embodiment, a text information extraction method disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the text information extraction method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities. In some possible implementations, the text information extraction method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0048] See also Figure 1 FIG. 1 is a flowchart of a text information extraction method provided by an embodiment of the present disclosure. The method is applied to a service platform. The method includes steps S101 to S103, wherein:

[0049] S101: Acquire a video stream including a screen displayed by a vehicle-mounted device; wherein the video stream is acquired by collecting the displayed screen during a vehicle test.

[0050] In the disclosed embodiment, a camera device may be pre-installed in the vehicle, wherein the camera device's field of view is the display screen of the vehicle-mounted device. During the vehicle test, the camera device may be used to capture the screen displayed by the vehicle-mounted device to obtain a video stream, which is then sent to the service platform.

[0051] Here, the vehicle testing process may be a process of testing the driving performance of the vehicle, wherein the driving performance test of the vehicle includes a performance test during vehicle driving and a performance test during vehicle stationary state. The vehicle testing scene is not specifically limited here, and all vehicle testing scenes that can capture the display screen are within the protection scope of this application.

[0052] S102: extracting text from each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text contents in each video frame and an initial text area of ​​the all text contents.

[0053] In the disclosed embodiment, text extraction can be performed on each video frame using OCR text extraction technology, thereby obtaining all text content in each video frame and a text region of all text content, which is recorded as an initial text region.

[0054] S103: extracting text information of interest from the initial text information of each of the video frames, and extracting target text content from the text information of interest.

[0055] Here, the text information of interest is associated with the test scenario of the vehicle, and therefore, the text information of interest is the text information associated with the test scenario of the vehicle in the initial text information. For example, the text information of interest may be the text information associated with the real-time driving speed and engine speed of the vehicle in the initial text information.

[0056] After the text information of interest is extracted, the required text content may be extracted from the text information of interest and recorded as the target text content.

[0057] In the embodiments of the present disclosure, the method for extracting the text information of interest from the initial text information can be determined based on the position of each video frame in the video stream and / or the type of each video frame; wherein the type of video frame includes a key frame or a non-key frame, and the key frame can be understood as a video frame whose display screen has changed, and the non-key frame can be understood as a video frame whose display screen has not changed.

[0058] Through the processing method described above, it is possible to continuously acquire the images displayed by the vehicle-mounted device and continuously parse the text information, thereby acquiring the required text content more efficiently, comprehensively and directly.

[0059] The above steps will be described in detail below in conjunction with specific implementation methods.

[0060] It can be seen from the above description that the service platform can obtain the video stream captured by the shooting device, then scale each video frame, and extract text from the scaled video frames.

[0061] By scaling the video frames in proportion, the image size can be effectively standardized so that different video frames have the same size reference during subsequent processing. For example, the pixels of the longest side can be adjusted to 2048 by scaling in proportion. The unified scaling operation not only makes it easier to correspond and compare the position information between adjacent frames, reducing the problem of inconsistent position information caused by image size differences, but also provides more stable and consistent input conditions for OCR recognition, thereby improving the accuracy and reliability of OCR recognition.

[0062] Here, text extraction can be performed on the video frames after equal scaling using OCR text extraction technology. OCR text extraction technology is a technology for extracting text information from images. The PP-OCRv3 model is used to extract text from the preprocessed video frames, thereby obtaining the specific content and corresponding location information of all texts in the video frames.

[0063] In addition, text extraction can be performed on each video frame through a text information extractor (for example, a convolutional neural network), so as to extract all the text content in the corresponding video frame and the initial text area of ​​the all text content.

[0064] After the initial text information is extracted, a method for extracting the text information of interest from the initial text information can be determined according to the position of each video frame in the video stream and / or the type of each video frame, wherein the position of the video frame includes the first video frame and the non-first video frame.

[0065] In a specific implementation, it can be determined whether the video frame is the first video frame (also called the initial frame); if it is determined to be the first video frame, the text information of interest is extracted from the initial text information of each video frame in the following manner, which specifically includes the following steps:

[0066] In the case where the video frame is an initial frame in the video stream, the initial text information of the initial frame is subjected to text classification according to the purpose of text extraction through a text classification model to obtain the text information of interest of the initial frame.

[0067] In the disclosed embodiment, a text classification model can be pre-trained according to the purpose of text extraction; wherein the text classification model can be a Transformer-based text classification model. In addition, other types of neural network models can also be selected. Here, the purpose of text extraction is associated with the test scenario of the vehicle. For example, the test scenario of the vehicle is a test on the vehicle's driving speed. At this time, the purpose of text extraction can be to extract text information associated with the vehicle's real-time driving speed or engine speed in the video frame.

[0068] By performing text classification processing on the video frames through the text classification model, text information that matches the purpose of text extraction can be classified, that is, the text information of interest.

[0069] For the initial frame, the specific content and corresponding location information of the text in the image can be obtained through OCR text extraction technology. Since a lot of text information is obtained, and most of the text information is unnecessary redundant information, that is, information that is irrelevant to the vehicle test scenario. Therefore, the technical solution of this application obtains the required text content and its text area from all text content through a trained Transformer-based text classification model, and determines the text content and text area as text information of interest.

[0070] If it is determined that it is not the first video frame, the text information of interest is extracted from the initial text information of each video frame in the following manner, which specifically includes the following steps:

[0071] First, when the video frame is a non-initial frame in the video stream, the intersection-and-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame is calculated;

[0072] Secondly, when the intersection-and-union ratio meets the threshold requirement, the text information of interest of the video frame is determined according to the text region of interest corresponding to the maximum intersection-and-union ratio and the text content of interest corresponding to the text region of interest.

[0073] In the disclosed embodiment, if the video frame is not the first video frame in the video stream, the text information of interest can be extracted from the initial text information of the video frame by matching text regions between adjacent frames. The text information of interest includes text content of interest and text regions of interest in the text content of interest.

[0074] In a specific implementation, first, each text region of interest of the previous video frame can be obtained; wherein the text region of interest is the text region of interest extracted from the initial text information of the previous video frame. Next, the intersection-and-union ratio between the initial text region and each text region of interest in the previous video frame can be calculated. For example, the intersection-and-union ratio between each initial text region and each text region of interest can be calculated. Therefore, for each initial text region, at least one intersection-and-union ratio can be calculated.

[0075] Afterwards, it can be determined whether the intersection-and-union ratio meets the threshold requirement. If it is determined that the threshold requirement is met, the maximum intersection-and-union ratio is calculated among at least one intersection-and-union ratio corresponding to each initial text area, and then the text area of ​​interest corresponding to the maximum intersection-and-union ratio and the text content of interest corresponding to the text area of ​​interest are determined as the text information of interest of the current video frame.

[0076] Here, you can use the following methods to determine whether the intersection-over-union ratio meets the threshold requirement:

[0077] Method 1: If it is determined that each group of IoUs has an IoU greater than a first threshold, then the IoUs meets the threshold requirement; wherein the IoUs between an initial text region and each text region of interest in the previous video frame corresponds to a group of IoUs.

[0078] Method 2: If it is determined that the number of target IoU groups in the multiple IoU groups is less than the second threshold, it is determined that the IoU meets the threshold requirement; wherein, there is no IoU greater than the first threshold in the target IoU group.

[0079] In the embodiment of the present disclosure, when the intersection-over-union ratio does not meet the threshold requirement, the initial text information of the non-initial frame can also be classified and processed according to the purpose of text extraction through a text classification model to obtain the text information of interest in the non-initial frame.

[0080] For the above method 1, if it is determined that each group of intersection-and-union ratios does not have an intersection-and-union ratio greater than the first threshold, it can be determined that the image displayed by the vehicle-mounted device has undergone a significant change, for example, the interface layout of the display interface of the vehicle-mounted device has undergone a significant change. At this time, it is no longer possible to continue to use the text area matching method to extract the text information of interest from the initial text information.

[0081] For the above method 2, if it is determined that the number of target intersection-over-union groups is greater than the second threshold, it can be determined that the number of text regions matching between adjacent video frames is small, and at this time, it can also be determined that the image displayed by the vehicle-mounted device has changed. At this time, it is also impossible to continue to use the text region matching method to extract the text information of interest from the initial text information.

[0082] On this basis, the initial text information of the non-initial frame can be classified according to the purpose of text extraction through a text classification model to obtain the text information of interest. For example, a trained Transformer-based text classification model can be used to obtain the required text content and its text area from all text content, and the text content and text area can be determined as the text information of interest in the non-initial frame.

[0083] For non-initial frames, the specific content and corresponding position information of the text in the image can be obtained through OCR text extraction technology. Similar to the initial frame, the extracted initial text information contains a large amount of unnecessary redundant information. Therefore, the technical solution disclosed in the present invention can remove redundant information in the initial text information by matching text regions between adjacent frames to determine the text information of interest, so that the desired text content can be obtained more efficiently. At the same time, by setting the intersection and union ratio to meet the threshold requirement, the text region matching method is adopted to avoid the problem of text region matching failure caused by large changes in the display screen.

[0084] In an optional implementation, the above step of calculating the intersection-and-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame specifically includes the following steps:

[0085] First, each region of interest in the previous video frame is enlarged;

[0086] Secondly, the intersection-over-union ratio between each initial text region and the enlarged text region of interest is calculated.

[0087] In the disclosed embodiment, for each text region of interest in the previous video frame, a certain degree of region expansion processing is performed, and for a text region of interest with a smaller area, the length and width are expanded to 1.1-1.3 times the original, and for a text region of interest with a larger area, the length and width are expanded to 1.4-1.7 times the original. This processing method can enhance the robustness of region matching.

[0088] Here, an area threshold may be preset, wherein the threshold value of the area threshold may be set according to actual needs and is not specifically limited here. An area whose area is greater than or equal to the area threshold is determined as a text area of ​​interest with a larger area, and an area whose area is less than the area threshold is determined as a text area of ​​interest with a smaller area.

[0089] For all the initial text regions obtained by OCR in the current video frame, the intersection-over-union (IOU) is calculated with the enlarged text region of interest. Then, the result of the intersection-over-union calculation determines the text information of interest in the current frame. For example, the text region of interest with the largest IOU can be taken as the text region of interest in the current video frame. After that, the text information of interest in the current video frame is subjected to the next string matching operation, and the selected text region of interest is passed to the next video frame for text region matching.

[0090] In an optional implementation, the above step of calculating the intersection-and-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame specifically includes the following steps:

[0091] First, determining affine transformation parameters between the video frame and the previous video frame;

[0092] Secondly, the video frame and the previous video frame are image registered according to the affine transformation parameters, and the intersection-over-union ratio between the initial text region and each text region of interest in the previous video frame after registration is calculated.

[0093] In the actual vehicle driving test process, due to the bumps during driving, the camera device will have a certain degree of shaking, which may cause the image position between adjacent frames to deviate, which has a great impact on the accuracy of text area matching. Therefore, the technical solution disclosed in the present invention adopts the image anti-shake technology based on the optical flow method to perform anti-shake processing on the video frame, and performs text area matching processing based on the video frame after anti-shake processing.

[0094] In a specific implementation, the affine transformation parameters between the current video frame and the previous video frame can be determined by the feature points. The previous frame image is aligned with the current frame image through the affine transformation matrix, thereby achieving the image anti-shake effect. Next, the text area of ​​the registered video frame and the previous video frame can be matched to obtain the text information of interest in the current video frame. For example, the intersection-and-union ratio between the initial text area after registration and each text area of ​​interest in the previous video frame can be calculated, and the text information of interest in the current video frame can be determined based on the intersection-and-union ratio.

[0095] Through this processing method, the position area of ​​the same content in the two frames of images can be guaranteed to be consistent, greatly improving the accuracy of area matching.

[0096] In the embodiment of the present disclosure, the step of determining the affine transformation parameters between the video frame and the previous video frame specifically includes:

[0097] First, a plurality of strong corner points are determined in the previous video frame;

[0098] Secondly, determining the mapping position of each of the strong corner points in the video frame based on the optical flow method;

[0099] Then, the affine transformation parameters are determined according to the positions of the multiple strong corner points in the previous video frame and the mapping positions.

[0100] Here, strong corner point detection is performed on the previous video frame to find strong corner points with good features on the image; then, the optical flow method is used to determine the new position (i.e., mapping position) of the strong corner points in the current video frame based on the strong corner points detected in the previous video frame, the current video frame, and the previous video frame; then, the affine transformation parameters between the two frames are estimated based on the mapping positions of the strong corner points detected in the previous video frame and the strong corner points in the current video frame; finally, the previous video frame is aligned with the current video frame through the affine transformation parameters, thereby achieving the image stabilization effect. Through this processing method, the position area of ​​the same content in the two frames of images can be guaranteed to be consistent, greatly improving the accuracy of regional matching.

[0101] In the embodiment of the present disclosure, if it is determined that it is not the first video frame, the text information of interest may be extracted from the initial text information of each video frame in the following manner, which specifically includes the following steps:

[0102] The image similarity between the current video frame and the previous video frame is calculated; if it is determined that the similarity is greater than a preset threshold, the text information of interest in the current video frame is determined by text region matching, wherein the text region matching method is as described above and will not be repeated here. If it is determined that the similarity is less than the preset threshold, the text classification model performs text classification on the initial text information of the initial frame according to the text extraction purpose to obtain the text information of interest.

[0103] By calculating image similarity, key frames and non-key frames in the video stream can be identified, thereby ensuring that the target text content can be extracted from each video frame more efficiently and accurately.

[0104] After extracting the text information of interest, the target text content can be extracted from the text information of interest, which specifically includes the following steps:

[0105] A character string matching process is performed on the text content in the text information of interest to obtain a matching result, and the matching result is determined as the target text content.

[0106] Here, after the text information of interest is extracted in the manner described above, all the text content of interest in the text information of interest can be determined as the target text content. In addition, only part of the text in the entire text content of interest may be required, for example, numbers, specific characters, etc. Based on this, the technical solution disclosed in the present invention adopts a string regularization matching method to further extract the specific text content required in the text information of interest.

[0107] Through the above implementation, for the text information obtained by OCR, the string matching technology can be used to remove the redundant text content in the text information of interest, thereby quickly and accurately obtaining the desired text information.

[0108] like Figure 2 The flowchart shown is another method for extracting text information. Figure 2 As shown, the method specifically comprises the following steps:

[0109] S201: Obtain any video frame in a video stream.

[0110] S202: Perform image preprocessing on the video frame.

[0111] The service platform can obtain the video stream captured by the shooting device, then scale each video frame, and extract text from the scaled video frames.

[0112] By scaling the video frames in proportion, the image size can be effectively standardized so that different video frames have the same size reference during subsequent processing. For example, the pixels of the longest side can be adjusted to 2048 by scaling in proportion. The unified scaling operation not only makes it easier to correspond and compare the position information between adjacent frames, reducing the problem of inconsistent position information caused by image size differences, but also provides more stable and consistent input conditions for OCR recognition, thereby improving the accuracy and reliability of OCR recognition.

[0113] S203: extracting text from each video frame in the video stream to obtain initial text information; the initial text information includes all text contents in each video frame and an initial text area of ​​all text contents.

[0114] In the disclosed embodiment, text extraction can be performed on each video frame using OCR text extraction technology, thereby obtaining all text content in each video frame and a text region of all text content, which is recorded as an initial text region.

[0115] S204: Determine whether the currently processed video frame is an initial frame; if it is determined to be an initial frame, execute step S205; otherwise, execute step S207;

[0116] S205: Input the video frame into a text classification model for processing; wherein the text classification model performs text classification on the initial text information of the initial frame according to the purpose of text extraction to obtain text information of interest.

[0117] S206: Obtain the text information of interest output by the text classification model; here, the text information of interest of the initial frame also needs to be passed to step S207 for processing.

[0118] S207: Perform image stabilization processing on the current video frame according to the previous frame.

[0119] Specifically, strong corner point detection is performed on the previous video frame to find strong corner points with good features on the image; then, the optical flow method is used to determine the new position of the strong corner point in the current video frame (i.e., the mapping position) based on the strong corner points detected in the previous video frame, the current video frame and the previous video frame; thereafter, the affine transformation parameters between the two frames are estimated based on the strong corner points detected in the previous video frame and the mapping positions of the strong corner points in the current video frame; finally, the previous video frame is aligned with the current video frame through the affine transformation parameters, thereby achieving an image stabilization effect.

[0120] S208: Perform text region matching on the text region of interest of the previous frame and the initial text region of the current video frame to obtain the text region of interest of the current video frame.

[0121] Specifically, for each interesting text region in the previous video frame, a certain degree of region expansion processing is performed. For a smaller interesting text region, the length and width are expanded to 1.1-1.3 times the original, and for a larger interesting text region, the length and width are expanded to 1.4-1.7 times the original. This processing method can enhance the robustness of region matching.

[0122] For all the initial text regions obtained by OCR in the current video frame, the intersection over union (IOU) is calculated with the enlarged text region of interest. Then, the text region of interest with the largest IOU can be taken as the text region of interest in the current video frame.

[0123] S209: Determine the text information of interest in the current video frame according to the text region of interest in the current video frame and the text content in the text region of interest. Here, the text information of interest in the current video frame also needs to be transferred to step S207 for processing.

[0124] S210: Extracting target text content from the text information of interest.

[0125] Here, a character string matching process may be performed on the text content in the text information of interest to obtain a matching result, and the matching result is determined as the target text content.

[0126] S211: Return the target text content.

[0127] In the above-mentioned embodiment, during the vehicle testing process, by acquiring the video stream of the screen displayed by the vehicle-mounted device and extracting all the text information in each video frame in the video stream, the target text content is extracted from all the text information. It is possible to extract the desired text content for vehicle-mounted devices of different styles; at the same time, the text content displayed by the vehicle-mounted device can be continuously acquired, so that the required text content can be acquired more efficiently, comprehensively and directly.

[0128] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0129] Based on the same inventive concept, a text information extraction device corresponding to the text information extraction method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned text information extraction method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0130] Reference Figure 3 FIG. 1 is a schematic diagram of a text information extraction device provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition unit 10, a first text extraction unit 20, and a second text extraction unit 30; wherein:

[0131] The acquisition unit 10 is used to acquire a video stream containing a picture displayed by the vehicle-mounted device; wherein the video stream is acquired by collecting the displayed picture during the vehicle test process;

[0132] A first text extraction unit 20 is used to extract text from each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text contents in each video frame and an initial text area of ​​the all text contents;

[0133] The second text extraction unit 30 is used to extract text information of interest from the initial text information of each of the video frames, and to extract target text content from the text information of interest.

[0134] In the above-mentioned embodiment, during the vehicle testing process, by acquiring the video stream of the screen displayed by the vehicle-mounted device and extracting all the text information in each video frame in the video stream, the target text content is extracted from all the text information. It is possible to extract the desired text content for vehicle-mounted devices of different styles; at the same time, the text content displayed by the vehicle-mounted device can be continuously acquired, so that the required text content can be acquired more efficiently, comprehensively and directly.

[0135] In a possible implementation manner, the second text extraction unit is further used for:

[0136] In the case where the video frame is an initial frame in the video stream, the initial text information of the initial frame is subjected to text classification according to the purpose of text extraction through a text classification model to obtain the text information of interest of the initial frame.

[0137] In a possible implementation manner, the second text extraction unit is further used for:

[0138] When the video frame is a non-initial frame in the video stream, calculating an intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame;

[0139] When the intersection-and-union ratio meets the threshold requirement, the text information of interest of the video frame is determined according to the text region of interest corresponding to the maximum intersection-and-union ratio and the text content of interest corresponding to the text region of interest.

[0140] In a possible implementation manner, the second text extraction unit is further used for:

[0141] Performing region enlargement processing on each text region of interest in the previous video frame;

[0142] The intersection-over-union ratio between each of the initial text regions and the enlarged text region of interest is calculated.

[0143] In a possible implementation manner, the device is further used for:

[0144] When the intersection-over-union ratio does not meet the threshold requirement, the initial text information of the non-initial frame is classified according to the purpose of text extraction by using a text classification model to obtain the text information of interest of the non-initial frame.

[0145] In a possible implementation manner, the second text extraction unit is further used for:

[0146] Determining affine transformation parameters between the video frame and the previous video frame;

[0147] The video frame and the previous video frame are image registered according to the affine transformation parameters, and the intersection-over-union ratio between the initial text region and each text region of interest in the previous video frame after registration is calculated.

[0148] In a possible implementation manner, the second text extraction unit is further used for:

[0149] Determining a plurality of strong corner points in the previous video frame;

[0150] Determine the mapping position of each of the strong corner points in the video frame based on the optical flow method;

[0151] The affine transformation parameters are determined according to the positions of the multiple strong corner points in the previous video frame and the mapping positions.

[0152] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0153] Corresponds to Figure 1 The text information extraction method in the present disclosure also provides an electronic device 400, such as Figure 4 FIG. 4 is a schematic diagram of the structure of an electronic device 400 provided in an embodiment of the present disclosure, including:

[0154] Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including internal memory 421 and external memory 422; the internal memory 421 is also called internal memory, which is used to temporarily store the operation data in the processor 41 and the data exchanged with the external memory 422 such as a hard disk. The processor 41 exchanges data with the external memory 422 through the internal memory 421. When the electronic device 400 is running, the processor 41 communicates with the memory 42 through the bus 43, so that the processor 41 executes the following instructions:

[0155] Acquire a video stream containing a screen displayed by the vehicle-mounted device; wherein the video stream is obtained by collecting the displayed screen during the vehicle testing process;

[0156] Performing text extraction on each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​the all text content;

[0157] The text information of interest is extracted from the initial text information of each of the video frames, and the target text content is extracted from the text information of interest.

[0158] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the text information extraction method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0159] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the text information extraction method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0160] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0161] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0164] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0165] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A text information extraction method, characterized in that: include: Acquire a video stream containing a screen displayed by the vehicle-mounted device; wherein the video stream is obtained by collecting the displayed screen during the vehicle testing process; Performing text extraction on each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​the all text content; The text information of interest is extracted from the initial text information of each of the video frames, and the target text content is extracted from the text information of interest.

2. The method according to claim 1, characterized in that The extracting the text information of interest from the initial text information of each of the video frames comprises: In the case where the video frame is an initial frame in the video stream, the initial text information of the initial frame is subjected to text classification according to the purpose of text extraction through a text classification model to obtain the text information of interest of the initial frame.

3. The method according to claim 1 or 2, characterized in that: The extracting the text information of interest from the initial text information of each of the video frames comprises: When the video frame is a non-initial frame in the video stream, calculating an intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame; When the intersection-and-union ratio meets the threshold requirement, the text information of interest of the video frame is determined according to the text region of interest corresponding to the maximum intersection-and-union ratio and the text content of interest corresponding to the text region of interest.

4. The method according to claim 3, characterized in that The calculating the intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame includes: Performing region enlargement processing on each text region of interest in the previous video frame; The intersection-over-union ratio between each of the initial text regions and the enlarged text region of interest is calculated.

5. The method according to claim 3, characterized in that: The method further comprises: When the intersection-over-union ratio does not meet the threshold requirement, the initial text information of the non-initial frame is classified according to the purpose of text extraction by using a text classification model to obtain the text information of interest of the non-initial frame.

6. The method according to claim 3, characterized in that The calculating the intersection-over-union ratio between each initial text region in the initial text information and each text region of interest in the previous video frame includes: Determining affine transformation parameters between the video frame and the previous video frame; The video frame and the previous video frame are image registered according to the affine transformation parameters, and the intersection-over-union ratio between the initial text region and each text region of interest in the previous video frame after registration is calculated.

7. The method according to claim 6, characterized in that The determining of the affine transformation parameters between the video frame and the previous video frame comprises: Determining a plurality of strong corner points in the previous video frame; Determine the mapping position of each of the strong corner points in the video frame based on the optical flow method; The affine transformation parameters are determined according to the positions of the multiple strong corner points in the previous video frame and the mapping positions.

8. A text information extraction device, characterized in that: include: An acquisition unit, used to acquire a video stream containing a picture displayed by the vehicle-mounted device; wherein the video stream is acquired by collecting the displayed picture during the vehicle test process; A first text extraction unit, configured to extract text from each video frame in the video stream to obtain initial text information; wherein the initial text information includes all text content in each video frame and an initial text area of ​​the all text content; The second text extraction unit is used to extract text information of interest from the initial text information of each of the video frames, and to extract target text content from the text information of interest.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the text information extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the text information extraction method according to any one of claims 1 to 7 are implemented.

Images