Document generation method, device and electronic device
By identifying the missing locations of incomplete documents on electronic devices and collecting keyframe images, the problem of incomplete document content is solved, and the flexibility and intelligence of generating complete documents are achieved, improving the user experience.
Patent Information
- Application Number
- CN202210038800.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-13
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-01-13
AI Technical Summary
When existing image recognition technology generates editable documents, if the document content is incomplete and the missing parts cannot be effectively collected, resulting in insufficient user experience and flexibility.
By acquiring real-time images on electronic devices, identifying missing locations of unfinished documents, triggering mobile prompt information, prompting users to mobile devices to collect missing parts, obtain multiple keyframe images, stitching them into complete document images, and generating editable documents .
It improves the flexibility and intelligence of the image recognition process to generate documents, ensures the integrity of the generated documents, and improves the user experience.
Smart Images

Figure CN114429108B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of electronic devices, and more specifically, to a document generation method, device and electronic device. Background Art
[0002] With the development of image recognition technology, electronic devices can recognize the content in the acquired image and then store the recognition results. However, the related recognition process is still not flexible enough and the user experience needs to be improved. Summary of the invention
[0003] In view of the above problems, the present application proposes a document generation method, device and electronic device to improve the above problems.
[0004] In a first aspect, the present application provides a document generation method, which is applied to an electronic device, and the method includes: starting to acquire a real-time image captured by the electronic device; in response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to a missing position of the incomplete captured document, the movement prompt information being used to prompt a user to move the electronic device to capture text content corresponding to the missing position; acquiring multiple key frame images, the multiple key frame images including a real-time image captured by the electronic device at a starting position of movement, and at least one image selected from real-time images captured by the electronic device during movement; obtaining a complete document image based on the multiple key frame images; and generating a corresponding editable document according to the document content in the complete document image.
[0005] In a second aspect, the present application provides a document generation device that runs on an electronic device, the device comprising: an image acquisition unit, used to start acquiring a real-time image captured by the electronic device; an image recognition unit, used to trigger movement prompt information corresponding to a missing position of the incomplete captured document in response to identifying an incomplete captured document from the acquired real-time image, the movement prompt information being used to prompt a user to move the electronic device to capture text content corresponding to the missing position; a key frame acquisition unit, used to acquire multiple key frame images, the multiple key frame images including a real-time image captured by the electronic device at a starting position of movement, and at least one image selected from real-time images captured by the electronic device during movement; a document image generation unit, used to obtain a complete document image based on the multiple key frame images; and a document generation unit, used to generate a corresponding editable document based on the document content in the complete document image.
[0006] In a third aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.
[0007] In a fourth aspect, the present application provides a computer-readable storage medium storing a program code executable by a processor, wherein the computer-readable storage medium includes the stored program code, wherein the above method is executed when the program code is run.
[0008] The document generation method, device and electronic device provided by the present application can trigger movement prompt information corresponding to the missing position of the incompletely collected document in response to the incompletely collected document being identified from the acquired real-time image after starting to acquire the real-time image captured by the electronic device, so as to prompt the user to move the electronic device to collect the text content corresponding to the missing position. Then, the real-time image captured by the electronic device at the starting position of the movement and at least one image selected from the real-time images captured by the electronic device during the movement are used as multiple key frame images, and then a complete document image is obtained based on the multiple key frame images, and a corresponding editable document is generated according to the document content in the complete document image.
[0009] Therefore, through the above method, when an editable document needs to be generated, if the document collected by the electronic device is an incomplete document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0011] Figure 1 A schematic diagram of an application scenario of a document generation method proposed in an embodiment of the present application is shown;
[0012] Figure 2 A schematic diagram showing another application scenario of a document generation method proposed in an embodiment of the present application is shown;
[0013] Figure 3A flowchart of a document generation method proposed in an embodiment of the present application is shown;
[0014] Figure 4 A schematic diagram of collecting a complete document proposed in an embodiment of the present application is shown;
[0015] Figure 5 A schematic diagram of an incomplete collection document proposed in an embodiment of the present application is shown;
[0016] Figure 6 A schematic diagram showing another incomplete collection document proposed in an embodiment of the present application is shown;
[0017] Figure 7 A schematic diagram showing another mobile prompt information proposed in the present application;
[0018] Figure 8 A flowchart of a document generation method proposed in another embodiment of the present application is shown;
[0019] Fig. 9 A flowchart of a document generation method proposed in another embodiment of the present application is shown;
[0020] Fig.10 A flowchart of a document generation method proposed in another embodiment of the present application is shown;
[0021] Fig.11 A schematic diagram showing multiple key frame images in an embodiment of the present application is shown;
[0022] Fig.12 A schematic diagram of a complete document image obtained by splicing in an embodiment of the present application is shown;
[0023] Fig.13 A structural block diagram of another document generation device proposed in the present application is shown;
[0024] Fig.14 A structural block diagram of an electronic device for executing a document generation method according to an embodiment of the present application is shown;
[0025] Fig.15 It is a storage unit of an embodiment of the present application for storing or carrying program codes for implementing the document generation method according to an embodiment of the present application. DETAILED DESCRIPTION
[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0027] With the development of image recognition technology, electronic devices can recognize the content in the acquired image and then store the recognized results. However, the related recognition process is still not flexible enough and the user experience needs to be improved. For example, in some cases, the user expects to generate an editable document by operating the electronic device for image recognition. However, the physical document in the image content captured by the electronic device may not be complete, so the editable document cannot be generated well.
[0028] Therefore, the inventor proposes a document generation method, device and electronic device in the present application. After starting to acquire the real-time image captured by the electronic device, in the case of identifying an incomplete captured document based on the real-time image, the moving prompt information corresponding to the missing position of the incomplete captured document can be triggered in response to identifying the incomplete captured document from the acquired real-time image, so as to prompt the user to move the electronic device to capture the text content corresponding to the missing position. Then, the real-time image captured by the electronic device at the starting position of the movement and at least one image selected from the real-time images captured by the electronic device during the movement are used as multiple key frame images, and then a complete document image is obtained based on the multiple key frame images, and a corresponding editable document is generated according to the document content in the complete document image.
[0029] Therefore, through the above method, when an editable document needs to be generated, if the document collected by the electronic device is an incomplete document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus improving the user experience.
[0030] The following first introduces the application scenarios involved in the embodiments of the present application.
[0031] In the embodiment of the present application, the document generation method provided can be executed by an electronic device. In this way, all steps in the document generation method provided by the embodiment of the present application can be executed by a processor of the electronic device. For example, Figure 1As shown, the image acquisition device (camera) of the electronic device 100 can acquire real-time images and transmit the acquired real-time images to the processor, so that the processor can execute the document generation method provided in the embodiment of the present application.
[0032] Furthermore, the document generation method provided in the embodiment of the present application can also be performed by the electronic device and the server in collaboration. In this mode of being performed by the electronic device and the server in collaboration, some steps in the document generation method provided in the embodiment of the present application are performed by the electronic device, while other steps are performed by the server.
[0033] For example, Figure 2 As shown, the electronic device 100 can execute the document generation method including: starting to acquire the real-time image captured by the electronic device. Then, the server 200 executes, in response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to the missing position of the incomplete captured document, the movement prompt information is used to prompt the user to move the electronic device to capture the text content corresponding to the missing position; acquiring multiple key frame images, the multiple key frame images include the real-time image captured by the electronic device at the starting position of the movement, and at least one image selected from the real-time images captured by the electronic device during the movement; obtaining a complete document image based on the multiple key frame images; generating a corresponding editable document according to the document content in the complete document image. Then the server 200 returns the generated editable document to the electronic device 100.
[0034] It should be noted that in this method of collaborative execution by the electronic device and the server, the steps respectively executed by the electronic device and the server are not limited to the methods introduced in the above examples. In actual applications, the steps respectively executed by the electronic device and the server can be dynamically adjusted according to actual conditions.
[0035] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0036] See also Figure 3 The present application provides a document generation method, which is applied to an electronic device, and the method comprises:
[0037] S110: Start acquiring the real-time image collected by the electronic device.
[0038] In an embodiment of the present application, the real-time image is an image captured in real time after the camera of the electronic device is started. Optionally, the real-time image captured by the electronic device can also be understood as the video content captured by the electronic device in real time. Among them, the document generation method provided in the embodiment of the present application can be executed by a camera program in the electronic device, or it can be executed by a program installed in the electronic device specifically for generating documents. In the case where the document generation method provided in the embodiment of the present application is executed by a camera program in the electronic device, the camera program can start to acquire the real-time image captured by the electronic device after being switched to the document generation mode. In addition, in the case where the document generation method provided in the embodiment of the present application is executed by a program installed in the electronic device specifically for generating documents, the program specifically for generating documents can call the camera of the electronic device after startup to start image acquisition, and start to acquire the real-time image captured by the electronic device.
[0039] S120: In response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to a missing position of the incomplete captured document, wherein the movement prompt information is used to prompt a user to move the electronic device to capture text content corresponding to the missing position.
[0040] After obtaining the real-time image captured by the electronic device, the content in the real-time image can be identified to identify whether there is an incompletely captured document in the real-time image. Among them, the incompletely captured document can be understood as a document whose contents are not all captured by the electronic device at one time. Figure 4 As shown, in the real-time image display area 10, a document 11 is displayed, wherein all the contents of the document 11 are collected by the electronic device, and then the document 11 can be understood as a complete collection document. Figure 5 As shown, in Figure 5 If a document 12 is displayed in the real-time image display area 10, but the document 12 is not completely captured by the electronic device, the electronic device will determine that an incompletely captured document is recognized.
[0041] In an embodiment of the present application, a pre-trained image classifier can be used to identify the acquired real-time image to identify whether it includes an incomplete collected document. The image classifier can be implemented using a CNN (such as ResNet, MobileNet, etc.) network, or can be implemented using a ViT, RNN, etc. network, which is not limited in the embodiment of the present application. The trained image classifier can identify incomplete collected documents in various missing positions. The missing position may include the upper left part of the missing document, the upper side of the missing document, the upper right part of the missing document, the right side of the missing document, the lower right part of the missing document, the lower side of the missing document, the lower left part of the missing document, the left side of the missing document, etc. For example, Figure 5 The missing position of the incomplete captured document shown in is the right side of the document. Figure 6 As shown, in Figure 6 The missing position of the incomplete captured document 13 shown in FIG. 1 is the sitting side of the document.
[0042] In the process of training the image classifier, the acquired training data may include document images corresponding to 9 labels. The 9 labels include the labels of the 8 missing positions and the labels of the complete captured documents. After the initial classifier is trained based on the training data, the obtained image classifier can identify whether the image (for example, the aforementioned real-time image) includes a complete captured document and an incomplete captured document, and can also identify the missing position corresponding to the identified incomplete captured document.
[0043] The moving prompt information may be generated based on the missing position of the identified incomplete captured document. For example, if the missing position is on the right side of the document, the moving prompt information generated is to prompt the user to move the electronic device to the right. For another example, if the missing position is on the lower left part of the document, the moving prompt information generated is to prompt the user to move the electronic device to the lower left part. For example, Figure 7 As shown, when the missing position is identified as the right side of the document, it can be displayed Figure 7 The content shown in is the moving prompt information of "Please move the device to the right".
[0044] S130: Acquire a plurality of key frame images, wherein the plurality of key frame images include a real-time image captured by the electronic device at a moving starting position, and at least one image selected from real-time images captured by the electronic device during movement.
[0045] In the embodiment of the present application, the key frame image is an image used to generate a complete document image. As an example, the real-time image captured by the electronic device at the starting position of the movement is image a, and the real-time images captured by the electronic device during the movement include image b, image c, image d, image e, and image f. Optionally, image c, image d, and image f captured during the movement can be used as key frame images, and the multiple key frame images finally obtained include image a, image c, image d, and image f.
[0046] It should be noted that when the electronic device is performing image acquisition, the distance between the electronic device and the target object will affect the size of the target object in the image captured by the electronic device. The size can be understood as the proportion of the target object in the image. For example, the greater the distance between the target object and the electronic device, the smaller the size of the target object in the image captured by the electronic device, and correspondingly, the smaller the proportion of the target object in the captured image. Based on the above situation, when the electronic device recognizes that an incomplete captured document has been captured, if the distance between the electronic device and the physical document corresponding to the incomplete captured document is large enough, the electronic device can directly capture the complete physical document by performing translation. Specifically, directly capturing a complete physical document can be understood as the entire content of the physical document can be directly displayed in one frame of image captured by the electronic device, for example, Figure 4 The entire contents of the document 11 shown are directly displayed in a frame of image captured by the electronic device. However, when the distance between the electronic device and the physical document corresponding to the incompletely captured document is small, no matter how the electronic device is translated, only a certain part of the document can be captured, and thus the electronic device cannot completely capture all parts of the document at one time.
[0047] Then as a way, the electronic device can obtain multiple key frame images when it is detected that the entire content corresponding to the incompletely collected document cannot be directly displayed in the image display area of the electronic device by translating the electronic device. If it is detected that the entire content corresponding to the incompletely collected document can be directly displayed in the image display area of the electronic device by translating the electronic device, the complete document image can be directly obtained from the real-time image collected by the electronic device during the movement. In this way, the acquired real-time image including the complete document can be used as the complete document image. In this way, even if the physical document that needs to be collected is too large to be collected by the electronic device at one time, an image including all the contents in the physical document can still be obtained, thereby solving the problem that the page of the document to be scanned (physical document) is too large and the preview interface of the electronic device for displaying the real-time image cannot be fully covered.
[0048] S140: Obtaining a complete document image based on the multiple key frame images.
[0049] It should be noted that in the embodiment of the present application, the complete document image can be understood as an image that includes all the contents of the physical document corresponding to the incomplete collected document. Although a single key frame image does not include all the contents of the physical document corresponding to the incomplete collected document, each of the multiple key frame images can include part of the content of the physical document, and the part of the content of the physical document included in each key frame is different, so that the partial content of the physical document included in each of the multiple key frame images can be combined to include all the contents of the physical document corresponding to the incomplete collected document. Exemplarily, if all the contents of the physical document corresponding to the incomplete collected document identified by the real-time image include document content A, document content B, document content C and document content D. Correspondingly, the multiple key frame images may include key frame image P1, key frame image P2 and key frame image P3. Among them, key frame image P1 may include document content A, key frame image P2 may include document content B and document content C, and key frame image P3 may include document content D. It can be seen from this example that although key frame image P1, key frame image P2 and key frame image P3 each do not include all document content in the physical document, then the key frame image P1, key frame image P2 and key frame image P3 can be combined to include all document content in the physical document. Therefore, a complete document image can be obtained based on the key frame image P1, key frame image P2 and key frame image P3.
[0050] S150: Generate a corresponding editable document according to the document content in the complete document image.
[0051] Among them, in the embodiment of the present application, the editable document can be understood as a document whose content can be edited. For example, the editable documents in the embodiment of the present application include documents in formats such as doc, xls, and ppt. As a way, the corresponding editable document is generated according to the document content in the complete document image, including: obtaining the type of document content included in the complete document image; generating an editable document based on the type of document content included, wherein in the editable document, the editable type of each document content corresponds to the type of the document content. For example, the electronic device can recognize the content included in the complete document image by OCR (Optical Character Recognition) to identify the document content of the type of text, table, or picture in the complete document image, so that the document content of this type in the generated editable document is still the original type in the physical document.
[0052] The document generation method provided in this embodiment, after starting to acquire the real-time image captured by the electronic device, in the case of identifying an incomplete captured document based on the real-time image, in response to identifying the incomplete captured document from the acquired real-time image, triggering the movement prompt information corresponding to the missing position of the incomplete captured document, so as to prompt the user to move the electronic device to capture the text content corresponding to the missing position. Then, the real-time image captured by the electronic device at the starting position of the movement, and at least one image selected from the real-time image captured by the electronic device during the movement, are used as multiple key frame images, and then a complete document image is obtained based on the multiple key frame images, and a corresponding editable document is generated according to the document content in the complete document image. Thus, in the process of generating an editable document, if the document captured by the electronic device is an incomplete captured document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then a complete document image can be generated, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus also improving the user experience.
[0053] See also Figure 8 The present application provides a document generation method, which is applied to an electronic device, and the method comprises:
[0054] S210: Start acquiring the real-time image collected by the electronic device.
[0055] S220: In response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to a missing position of the incomplete captured document, wherein the movement prompt information is used to prompt a user to move the electronic device to capture text content corresponding to the missing position.
[0056] S230: Using the real-time image captured by the electronic device at the moving starting position as the first key frame image.
[0057] S240: If the electronic device starts to move, whenever the movement of the electronic device meets the specified conditions, the currently captured real-time image is used as the second key frame image until the key frame image is captured. As a method, the specified conditions include: the movement duration of the electronic device meets the specified duration; or the movement distance of the electronic device meets the specified length. In the embodiment of the present application, the electronic device can calculate the movement distance of the electronic device by the optical flow method.
[0058] As a method, before acquiring multiple key frame images, it also includes: taking the missing position of the incomplete captured document as the first missing position; after taking the real-time image captured by the electronic device at the moving starting position as the first key frame image, it also includes: if an incomplete captured document is identified from the captured real-time image and the missing position of the incomplete captured document is the second missing position, determining to end capturing the key frame image; wherein, the second missing position is relative to the first exact position.
[0059] As described in the above embodiment, in the embodiment of the present application, the missing position of the document content can be the 8 positions shown above. Among them, during the translation of the electronic device, if the missing position of the incompletely collected document collected by the electronic device changes from a certain missing position to a relative missing position, it means that the electronic device has performed image collection of all document contents in the physical document corresponding to the incompletely collected document.
[0060] S250: Use the first key frame image and the second key frame image as multiple key frame images, wherein the multiple key frame images include a real-time image captured by the electronic device at a moving starting position, and at least one image selected from real-time images captured by the electronic device during the movement.
[0061] S260: Obtaining a complete document image based on the multiple key frame images.
[0062] S270: Generate a corresponding editable document according to the document content in the complete document image.
[0063] The present embodiment provides a document generation method, so that in the process of generating an editable document, if the document collected by the electronic device is an incomplete document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus also improving the user experience. Moreover, in the present embodiment, after the prompt information for moving is triggered, the multiple frames of images collected by the electronic device at the start position of the movement and during the movement can be used as key frames, so that the complete document image can be spliced more accurately. Furthermore, in the present embodiment, the relative missing position can be used as the end point of the movement, so that the electronic device can more accurately determine the end point of the movement.
[0064] See also Fig. 9 The present application provides a document generation method, which is applied to an electronic device, and the method comprises:
[0065] S310: Start acquiring the real-time image collected by the electronic device.
[0066] S320: In response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to the missing position of the incomplete captured document, wherein the movement prompt information is used to prompt the user to move the electronic device to capture text content corresponding to the missing position.
[0067] S330: Acquire a plurality of key frame images, wherein the plurality of key frame images include a real-time image captured by the electronic device at a moving starting position, and at least one image selected from real-time images captured by the electronic device during the movement.
[0068] S340: Based on the acquisition order of the multiple key frame images, the multiple key frame images are sequentially spliced to obtain a complete document image.
[0069] As a method, the multiple key frame images are sequentially spliced based on the acquisition order of the multiple key frame images to obtain a complete document image, including: obtaining feature points corresponding to each of the multiple key frame images. The feature points of the key frame images can be described by feature descriptors such as SURF, SIFT, and ORB.
[0070] The feature points of each two adjacent key frame images are feature matched to obtain a set of matching feature points corresponding to each two adjacent key frame images. Image registration is performed based on the set of matching feature points corresponding to each two adjacent key frame images to obtain a transformation matrix corresponding to each two adjacent key frame images. The transformation matrix can be used to convert the same feature points in the corresponding two key frame images to the same coordinate system, and then the multiple key frame images can be spliced in sequence based on the transformation matrix corresponding to each two adjacent key frame images to obtain a complete document image.
[0071] Optionally, in an embodiment of the present application, based on the transformation matrix corresponding to every two adjacent key frame images captured, the multiple key frame images are sequentially stitched together to obtain a complete document image, which may include: starting from the second key frame image in the acquisition order, taking each key frame image in turn as the image to be stitched for the current image stitching, wherein, in the current image stitching process, based on the transformation matrix between the image to be stitched and the reference image in the current image stitching process, the image to be stitched is stitched into the reference image to obtain a stitched image output by the current image stitching, and the stitched image output by the current image stitching is used as the reference image for the next image stitching, so as to stitch the multiple key frame images to obtain a complete document image; wherein, the reference image corresponding to the first image stitching is the first key frame image in the acquisition order, the image to be stitched corresponding to each stitching process is the key frame image stitched in the current stitching process, and the transformation matrix between the image to be stitched and the reference image in the current image stitching process is the transformation matrix between the image to be stitched in the current image stitching process and the key frame image adjacent to the previous one in the acquisition order.
[0072] Exemplarily, the key frame images used to obtain the complete document image include key frame image P1, key frame image P2 and key frame image P3. The acquisition order of key frame image P1 is the first, the acquisition order of key frame image P1 is the second, and the acquisition order of key frame image P3 is the third. Then, in the first image stitching process, key frame image P1 can be used as the reference image, key frame image P2 can be used as the image to be stitched, and based on the transformation matrix between key frame image P1 and key frame image P2, key frame image P2 can be stitched into key frame image P1 to obtain the stitching image corresponding to the first image stitching.
[0073] In the second image stitching process, the stitching image corresponding to the first image stitching can be used as the reference image, and the key frame image P3 can be used as the image to be stitched. For the key frame image P3, the key frame image that is adjacent to the previous key frame image in the acquisition order is the key frame image P2. In the second image stitching process, the key frame image P2 can be stitched to the stitching image corresponding to the first image stitching based on the transformation matrix between the key frame image P2 and the key frame image P3, thereby obtaining a complete document image.
[0074] Optionally, the method of using the stitched image output by the current image stitching as the reference image for the next image stitching includes: obtaining a target area in the stitched image output by the current image stitching, the target area being an area included in both the image to be stitched and the reference image in the current image stitching process; obtaining replacement content corresponding to the target area by weighted averaging based on pixels of the image to be stitched in the current image stitching process corresponding to the target area and pixels of the reference image in the current image stitching process corresponding to the target area; replacing content of the stitched image output by the current image stitching corresponding to the target area with the replacement content to obtain a newly output stitched image, and using the newly output stitched image as the reference image for the next image stitching.
[0075] S350: Generate a corresponding editable document according to the document content in the complete document image.
[0076] The document generation method provided in this embodiment allows, through the above-mentioned method, in the process of generating an editable document, if the document collected by the electronic device is an incompletely collected document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus also improving the user experience. In addition, in this embodiment, in the process of splicing multiple key frame images, the transformation matrix between each two adjacent collected key frame images can be obtained by obtaining the feature points of the key frame images and then matching the feature points based on the feature points of the images, so that the splicing between the key frame images can be realized based on the transformation matrix between each two adjacent collected key frame images, so as to improve the accuracy of determining the splicing points, and also make it more likely that the document content represented by the complete document image finally spliced is consistent with the content of the physical document.
[0077] See also Fig.10 The present application provides a document generation method, which is applied to an electronic device, and the method comprises:
[0078] S410: Start acquiring the real-time image collected by the electronic device.
[0079] S420: In response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to a missing position of the incomplete captured document, wherein the movement prompt information is used to prompt a user to move the electronic device to capture text content corresponding to the missing position.
[0080] S430: Acquire a plurality of key frame images, wherein the plurality of key frame images include a real-time image captured by the electronic device at a moving starting position, and at least one image selected from real-time images captured by the electronic device during the movement.
[0081] S440: Obtaining a complete document image based on the multiple key frame images.
[0082] S450: Performing specified format processing on the complete document image to obtain a processed complete document image.
[0083] S460: Generate a corresponding editable document according to the document content in the processed complete document image.
[0084] The specified format processing includes at least one of the following processing methods: cropping the blank area between the document content and the document edge and correcting the complete document image.
[0085] The blank area is an area where no content is displayed, for example, an area where no text, table, or image is displayed.
[0086] The document generation method provided in this embodiment allows, through the above method, in the process of generating an editable document, if the document collected by the electronic device is an incompletely collected document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus also improving the user experience. In addition, in this embodiment, after obtaining the complete document image, the complete document image can be first processed in a specified format, which is conducive to obtaining a complete document image that is easier to identify.
[0087] The document generation method involved in the embodiment of the present application is described below by an example. Fig.11 As shown, the key frame image first in the acquisition sequence includes an incomplete acquisition document, and the incomplete acquisition document includes document content A in the complete document. It can be understood that when the incomplete acquisition document shown in the key frame image first in the acquisition sequence is identified, the electronic device can trigger the movement prompt information in the direction indicated by the arrow, and the electronic device can collect the document content A in the process of movement. Fig.11As shown in the second key frame image of the acquisition sequence, in the second key frame image of the acquisition sequence, since the electronic device has moved a certain distance in the direction indicated by the arrow, a part of the document content A (part A1) can no longer be captured by the electronic device, and the document content in the complete document included in the second key frame image of the acquisition sequence is document content A2 and document content B. Similarly, the document content in the complete document included in the third key frame image of the acquisition sequence is document content B and document content C. Then based on Fig.11 After the three key frame images shown in FIG. 1 are stitched together, the resulting complete document image can be shown as Fig.12 shown.
[0088] See also Fig.13 The document generation device 500 provided in the present application is run on an electronic device, and the device 500 includes:
[0089] The image acquisition unit 510 is used to start acquiring the real-time image collected by the electronic device.
[0090] The image recognition unit 520 is used to trigger movement prompt information corresponding to the missing position of the incomplete captured document in response to identifying an incomplete captured document from the acquired real-time image, and the movement prompt information is used to prompt the user to move the electronic device to capture the text content corresponding to the missing position.
[0091] The key frame acquisition unit 530 is used to acquire multiple key frame images, wherein the multiple key frame images include a real-time image captured by the electronic device at the starting position of the movement, and at least one image selected from the real-time images captured by the electronic device during the movement.
[0092] A document image generating unit 540, configured to obtain a complete document image based on the multiple key frame images;
[0093] The document generating unit 550 is configured to generate a corresponding editable document according to the document content in the complete document image.
[0094] As a method, the key frame acquisition unit 530 is specifically used to use the real-time image collected by the electronic device at the starting position of the movement as the first key frame image; if the electronic device starts to move, each time the movement of the electronic device meets the specified condition, the currently collected real-time image is used as the second key frame image until the key frame image is collected; the first key frame image and the second key frame image are used as multiple key frame images. Optionally, the specified condition includes: the movement duration of the electronic device meets the specified duration; or the movement distance of the electronic device meets the specified length.
[0095] As a method, the key frame acquisition unit 530 is also used to use the missing position of the incomplete captured document as the first missing position. If an incomplete captured document is identified from the captured real-time image and the missing position of the incomplete captured document is the second missing position, it is determined to end the acquisition of the key frame image; wherein the second missing position is relative to the first exact position.
[0096] The document image generation unit 540 is specifically used to sequentially stitch the multiple key frame images based on the acquisition order of the multiple key frame images to obtain a complete document image. Optionally, the document image generation unit 540 is specifically used to obtain the feature points corresponding to each of the multiple key frame images; perform feature matching on the feature points of each two adjacent key frame images to obtain a set of matching feature points corresponding to each two adjacent key frame images; perform image registration based on the set of matching feature points corresponding to each two adjacent key frame images to obtain a transformation matrix corresponding to each two adjacent key frame images; and sequentially stitch the multiple key frame images based on the transformation matrix corresponding to each two adjacent key frame images to obtain a complete document image.
[0097] Optionally, the document image generation unit 540 is specifically used to start from the second key frame image in the acquisition sequence, and sequentially use each key frame image as the image to be spliced for the current image splicing, wherein, in the current image splicing process, based on the transformation matrix between the image to be spliced and the reference image in the current image splicing process, the image to be spliced is spliced into the reference image to obtain a spliced image output by the current image splicing, and the spliced image output by the current image splicing is used as the reference image for the next image splicing, so as to splice multiple key frame images to obtain a complete document image; wherein, the reference image corresponding to the first image splicing is the first key frame image in the acquisition sequence, the image to be spliced corresponding to each splicing process is the key frame image spliced in the current splicing process, and the transformation matrix between the image to be spliced and the reference image in the current image splicing process is the transformation matrix between the image to be spliced in the current image splicing process and the key frame image adjacent to the previous one in the acquisition sequence.
[0098] Optionally, the document image generation unit 540 is specifically used to obtain a target area in the stitched image output by the current image stitching process, wherein the target area is an area included in both the image to be stitched and the reference image in the current image stitching process; obtain replacement content corresponding to the target area by weighted averaging based on pixels of the image to be stitched in the current image stitching process corresponding to the target area and pixels of the reference image in the current image stitching process corresponding to the target area; replace the content of the stitched image output by the current image stitching process corresponding to the target area with the replacement content to obtain a newly output stitched image, and use the newly output stitched image as the reference image for the next image stitching.
[0099] The document generation unit 550 is specifically used to perform specified format processing on the complete document image to obtain a processed complete document image; generate a corresponding editable document according to the document content in the processed complete document image; wherein the specified format processing includes at least one of the following processing methods: cropping the blank area between the document content and the document edge and correcting the complete document image.
[0100] The document generation unit 550 is specifically used to obtain the type of document content included in the complete document image; generate an editable document based on the type of document content included, wherein in the editable document, the editable type of each document content corresponds to the type of the document content.
[0101] It should be noted that the device embodiments in the present application correspond to the aforementioned method embodiments, and the specific implementation principles of each unit in the device embodiments are similar to the principles in the aforementioned method embodiments. The specific contents in the device embodiments can be found in the method embodiments, and will not be repeated in the device embodiments.
[0102] The following will be combined Fig.14 An electronic device provided by the present application is described.
[0103] See also Fig.14 Based on the above-mentioned text processing method and device, the embodiment of the present application also provides another electronic device 200 that can execute the above-mentioned text processing method. The electronic device 200 includes one or more (only one is shown in the figure) processors 102, a memory 104 and a network module 106 that are coupled to each other. Among them, the memory 104 stores a program that can execute the content of the above-mentioned embodiment, and the processor 102 can execute the program stored in the memory 104.
[0104] Among them, the processor 102 may include one or more cores for processing data. The processor 102 uses various interfaces and lines to connect various parts of the entire electronic device 200, and executes various functions and processes data of the electronic device 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory 104, and calling data stored in the memory 104. Optionally, the processor 102 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 102 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 102, but may be implemented separately through a communication chip.
[0105] The memory 104 may include a random access memory (RAM) or a read-only memory (ROM). The memory 104 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 104 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the terminal 100 during use.
[0106] The network module 106 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with an audio playback device. The network module 106 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM) cards, memories, etc. The network module 106 may communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices via wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. For example, the network module 106 may exchange information with a base station.
[0107] Please refer to Fig.15 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 1100 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.
[0108] The computer readable storage medium 1100 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1100 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1100 has storage space for program code 1110 that executes any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1110 can be compressed, for example, in an appropriate form.
[0109] In summary, the document generation method, device and electronic device provided by the present application can trigger movement prompt information corresponding to the missing position of the incompletely collected document in response to the incompletely collected document being identified from the acquired real-time image after starting to acquire the real-time image captured by the electronic device, in order to prompt the user to move the electronic device to collect the text content corresponding to the missing position. Then, the real-time image captured by the electronic device at the starting position of the movement and at least one image selected from the real-time images captured by the electronic device during the movement are used as multiple key frame images, and then a complete document image is obtained based on the multiple key frame images, and a corresponding editable document is generated according to the document content in the complete document image.
[0110] Therefore, through the above method, when an editable document needs to be generated, if the document collected by the electronic device is an incomplete document (for example, the document content is missing), the corresponding prompt information can be triggered so that the electronic device can collect the missing part of the document content, and then generate a complete document image, thereby improving the flexibility and intelligence of the document generation process based on image recognition, and thus improving the user experience.
[0111] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0112] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0113] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.
[0115] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical feature diagrams therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A document generation method, characterized in that: Applied to electronic equipment, the method comprises: Start acquiring the real-time image collected by the electronic device; In response to identifying an incomplete captured document from the acquired real-time image, triggering movement prompt information corresponding to the missing position of the incomplete captured document, the movement prompt information is used to prompt a user to move the electronic device to capture text content corresponding to the missing position, and the movement prompt information is generated according to the identified missing position of the incomplete captured document; If it is detected that all contents corresponding to the incompletely captured document are directly displayed in the image display area of the electronic device by translating the electronic device, the real-time image including the complete document acquired during the translation process is used as the complete document image; If it is detected that the entire contents corresponding to the incompletely captured document cannot be directly displayed in the image display area of the electronic device by translating the electronic device, a plurality of key frame images are acquired, wherein the plurality of key frame images include a real-time image captured by the electronic device at a moving starting position, and at least one image selected from the real-time images captured by the electronic device during the movement; Obtaining a complete document image based on the multiple key frame images; A corresponding editable document is generated according to the document content in the complete document image.
2. The method according to claim 1, characterized in that The step of acquiring a plurality of key frame images comprises: Using the real-time image collected by the electronic device at the moving starting position as the first key frame image; If the electronic device starts to move, whenever the movement of the electronic device meets the specified condition, the currently captured real-time image is used as the second key frame image until the key frame image is captured; The first key frame image and the second key frame image are used as a plurality of key frame images.
3. The method according to claim 2, characterized in that The specified conditions include: The duration of movement of the electronic device meets the specified duration; or The moving distance of the electronic device meets the specified length.
4. The method according to claim 2, characterized in that: Before acquiring a plurality of key frame images, the method further includes: Taking the missing position of the incompletely collected document as the first missing position; The method further comprises: taking the real-time image collected by the electronic device at the moving starting position as the first key frame image; If an incomplete captured document is identified from the captured real-time image and the missing position of the incomplete captured document is the second missing position, it is determined to end the capturing of the key frame image; wherein the second missing position is relative to the first missing position.
5. The method according to claim 1, characterized in that The step of obtaining a complete document image based on the multiple key frame images includes: Based on the acquisition order of the multiple key frame images, the multiple key frame images are spliced in sequence to obtain a complete document image.
6. The method according to claim 5, characterized in that The step of sequentially splicing the multiple key frame images based on the acquisition order of the multiple key frame images to obtain a complete document image includes: Obtaining feature points corresponding to each of the multiple key frame images; Perform feature matching on the feature points of every two adjacent key frame images to obtain a set of matching feature points corresponding to every two adjacent key frame images; Performing image registration based on a set of matching feature points corresponding to each two adjacent key frame images to obtain a transformation matrix corresponding to each two adjacent key frame images; Based on the transformation matrix corresponding to every two adjacent key frame images, the multiple key frame images are sequentially spliced to obtain a complete document image.
7. The method according to claim 6, characterized in that The step of sequentially splicing the multiple key frame images based on the transformation matrix corresponding to each two adjacent key frame images to obtain a complete document image includes: Starting from the second key frame image in the acquisition sequence, each key frame image is sequentially used as an image to be stitched for current image stitching, wherein, in the current image stitching process, based on the transformation matrix between the image to be stitched and the reference image in the current image stitching process, the image to be stitched is stitched into the reference image to obtain a stitched image output by the current image stitching, and the stitched image output by the current image stitching is used as the reference image for the next image stitching, so as to stitch multiple key frame images to obtain a complete document image; Among them, the reference image corresponding to the first image stitching is the key frame image that is first in the acquisition order, the image to be stitched corresponding to each stitching process is the key frame image stitched in the current stitching process, and the transformation matrix between the image to be stitched in the current image stitching process and the reference image is the transformation matrix between the image to be stitched in the current image stitching process and the key frame image that is adjacent to the previous one in the acquisition order.
8. The method according to claim 7, characterized in that The step of using the stitched image outputted from the current image stitching as a reference image for the next image stitching comprises: Acquire a target area in the stitched image output by the current image stitching, wherein the target area is an area included in both the image to be stitched and the reference image in the current image stitching process; Obtain replacement content corresponding to the target area by performing weighted averaging based on pixels of the image to be stitched in the current image stitching process corresponding to the target area and pixels of the reference image in the current image stitching process corresponding to the target area; The content of the spliced image output from the current image splicing corresponding to the target area is replaced with the replacement content to obtain a newly output spliced image, and the newly output spliced image is used as a reference image for the next image splicing.
9. The method according to claim 1, characterized in that: The generating a corresponding editable document according to the document content in the complete document image comprises: Performing specified format processing on the complete document image to obtain a processed complete document image; generating a corresponding editable document according to the document content in the processed complete document image; The specified format processing includes at least one of the following processing methods: cropping the blank area between the document content and the document edge and correcting the complete document image.
10. The method according to claim 1, characterized in that The generating a corresponding editable document according to the document content in the complete document image comprises: Acquire the type of document content included in the complete document image; An editable document is generated based on the types of the included document contents, wherein in the editable document, the editable type of each document content corresponds to the type of the document content.
11. A document generation device, characterized in that: Running on an electronic device, the device comprises: An image acquisition unit, used to start acquiring a real-time image captured by the electronic device; an image recognition unit, configured to trigger movement prompt information corresponding to a missing position of the incomplete captured document in response to recognizing an incomplete captured document from the acquired real-time image, wherein the movement prompt information is used to prompt a user to move the electronic device to capture text content corresponding to the missing position, and the movement prompt information is generated according to the identified missing position of the incomplete captured document; A key frame acquisition unit is used to, if it is detected that all contents corresponding to the incompletely collected document are directly and completely displayed in the image display area of the electronic device by translating the electronic device, use the real-time image including the complete document acquired during the translation process as the complete document image; if it is detected that all contents corresponding to the incompletely collected document cannot be directly and completely displayed in the image display area of the electronic device by translating the electronic device, acquire a plurality of key frame images, the plurality of key frame images including the real-time image acquired by the electronic device at the starting position of the movement, and at least one image selected from the real-time images acquired by the electronic device during the movement; A document image generating unit, configured to obtain a complete document image based on the plurality of key frame images; The document generating unit is used to generate a corresponding editable document according to the document content in the complete document image.
12. An electronic device, characterized in that: comprising one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 10.
13. A computer-readable storage medium storing a program code executable by a processor, characterized in that: The computer-readable storage medium includes stored program code, wherein when the program code is executed, the method according to any one of claims 1 to 10 is executed.
Citation Information
Patent Citations
Shooting method for splicing document images
CN102012629A
Image processing method and mobile terminal
CN108038825A
Multi-instance document key information extraction method and system
CN113536798A
Recognition and selection of discrete patterns within a scene or image
US10846561B1
Text stitching from multiple images
US20060013444A1
