End-cloud collaborative incremental optical character recognition method and electronic device

Abstract

The present disclosure relates to the technical field of optical character recognition, and discloses an end-cloud cooperative incremental optical character recognition method and an electronic device. A server receives key frame fragments and strip fragments carrying an in-session serial number from a scanning pen device end, the strip fragments carrying image strips aligned at the device end and a tail appended mark; when the serial number is equal to an expected serial number, the fragments are accepted and written into a replayable semantic fragment ledger; an incremental canvas is initialized according to the key frame fragments, and the image strips are appended to the tail of the effective area according to the tail appended mark; when a recognition trigger condition is met, stage recognition is performed on the effective area and a recognition trigger baseline is updated; in response to a rollback request carrying a checkpoint serial number, the ledger is truncated, the incremental canvas, the trigger baseline and the derived stage recognition results are emptied, the canvas is reconstructed by replaying the retained fragments, and a reinitiation serial number is returned. Thus, only the newly added content is uploaded to maintain consistency between the end and the cloud, and the system can be rolled back to a trusted checkpoint at a small cost.

Description

Technical Field

[0001] This disclosure relates to the field of optical character recognition technology, and in particular to edge-cloud collaborative incremental optical character recognition technology. Background Technology

[0002] The descriptions in this section are intended to provide background information for the implementation of this disclosure and should not be construed as an admission or implication that they constitute prior art.

[0003] Scanning pens typically use 2D image sensors to continuously capture local images of text at a high frame rate, with large overlapping areas between adjacent images. The device needs to estimate the scanning direction and horizontal and vertical displacement, sequentially stitching the non-overlapping portions of each frame into a long strip image, which is then input into an optical character recognition (OCR) model to obtain the text result. Due to limitations in processor, memory, power consumption, and network bandwidth, existing solutions typically obtain recognition results in the following ways: The entire stitching and recognition is completed on the device, but the local recognition model and long image processing consume significant computing and storage resources; the complete stitched image is uploaded to a server for recognition after scanning, but the user can only obtain the result after scanning, resulting in a significant delay; the original frames are uploaded frame by frame to the server for stitching and recognition, but the repeated pixels in adjacent frames cause the uploaded byte size to increase approximately linearly with the number of captured frames. Furthermore, common reliable transmission mechanisms ensure data packet delivery through sequence numbers, acknowledgments, and retransmissions, addressing data integrity at the network or transport layer. Summary of the Invention

[0004] This disclosure provides an edge-cloud collaborative incremental optical character recognition method and electronic device, which can solve the technical problem of how to ensure that the incremental canvas and stage recognition state of the server is consistent with the actual effective splicing state of the device when the scanning pen device only uploads new image content to the server, and how to restore the splicing state of the device at a small cost when it is unreliable.

[0005] According to one aspect of this disclosure, an edge-cloud collaborative incremental optical character recognition method is provided, applied to a server, including: The device receives semantic segments from the scanning pen device. The semantic segments carry in-session sequence numbers and include keyframe segments and strip segments. The strip segments also carry image strips and tail append markers. The image strips are newly added column images exported from the local splicing canvas by the scanning pen device and located between the uploaded width and the current effective width. The tail append markers are used to indicate that the image strips have been spatially aligned by the scanning pen device and should be appended to the tail boundary of the effective area of ​​the incremental canvas. When the in-session sequence number equals the expected sequence number, the semantic fragment is accepted, the accepted semantic fragment is written into the replayable semantic fragment ledger according to the in-session sequence number, and the expected sequence number is advanced. The incremental canvas and its effective area are initialized based on the keyframe segments; Based on the tail append mark, the image strip is appended to the tail boundary of the effective region of the incremental canvas, and the effective region is updated according to the width of the image strip; When the recognition trigger condition is met, phased optical character recognition is performed on the image located in the effective area of ​​the incremental canvas, and the recognition trigger baseline is updated. The recognition trigger baseline is the reference quantity on which the recognition trigger condition is determined. In response to a rollback request carrying a checkpoint sequence number from the scanning pen device, semantic fragments with in-session sequence numbers greater than the checkpoint sequence number in the semantic fragment ledger are deleted. The incremental canvas, the recognition trigger baseline, and the stage optical character recognition results derived from the deleted semantic fragments are cleared. The semantic fragments retained in the semantic fragment ledger are replayed in in-session sequence number order to reconstruct the incremental canvas. A retransmission start sequence number determined based on the checkpoint sequence number is returned to the scanning pen device so that the scanning pen device can retransmit the semantic fragments from the retransmission start sequence number.

[0006] In a preferred embodiment, the semantic segmentation also carries canvas generation; The step of accepting the semantic fragment when the in-session sequence number is equal to the expected sequence number includes: accepting the semantic fragment when the canvas generation carried by the semantic fragment is equal to the current canvas generation and the in-session sequence number is equal to the expected sequence number, and rejecting semantic fragments carrying canvas generations different from the current canvas generation. The method further includes: after completing the replay, establishing a new canvas generation different from the previous canvas generation as the current canvas generation, and returning the new canvas generation to the scanning pen device along with the replay start number.

[0007] In a preferred embodiment, the semantic segment also carries a predecessor state fingerprint; The method further includes: for each semantic segment received, calculating a new prefix state fingerprint based on the previous prefix state fingerprint, the current canvas generation, the session sequence number of the semantic segment, the segment type, the placement semantics, and the image payload digest, wherein the placement semantics include the placement parameters of the tail appending marker or the image stripe; The semantic segment is applied to the incremental canvas only if the predecessor state fingerprint carried by the semantic segment is equal to the current prefix state fingerprint. After the replay is completed, the checkpoint prefix state fingerprint obtained from the replay is returned to the scanning pen device as the reference fingerprint for the new canvas generation.

[0008] In a preferred embodiment, the recognition result of the staged optical character recognition carries a result version, which includes the canvas generation at the time of the recognition, the applied in-session sequence number, and the prefix state fingerprint. The method further includes: outputting the recognition result only when the canvas generation in the result version is equal to the current canvas generation and the prefix state fingerprint in the result version belongs to the current prefix state fingerprint sequence of the semantic shard ledger, and discarding recognition results that do not meet the aforementioned conditions.

[0009] In a preferred embodiment, the recognition trigger baseline includes the execution time of the previous stage optical character recognition and the content of the effective region when performing this recognition; The identification triggering condition is: the difference between the current time and the execution time reaches a time threshold, and the difference between the current content of the effective area and the content of the effective area when the identification is performed reaches a content growth threshold.

[0010] In a preferred embodiment, the method further includes: in response to the end of scanning, if a complete stitched image uploaded by the scanning pen device has been received, using the complete stitched image as the input for final-state optical character recognition; if the complete stitched image has not been received, using an image located within the effective area of ​​the incremental canvas as the input for final-state optical character recognition.

[0011] In a preferred embodiment, the semantic segmentation further includes a strip segment to be located, the strip segment to be located carrying an image strip and placement parameters but not carrying the tail append mark, the placement parameters including an additional width and a vertical displacement; The method further includes: appending a keyframe fragment received after the effective region of the incremental canvas has been initialized as a re-anchoring frame to the tail boundary of the effective region; and performing a validity reduction on the placement parameters carried by the strip fragment to be positioned before applying it to the incremental canvas, the validity reduction including taking a non-negative value for the added width and clamping the longitudinal displacement within a predetermined range.

[0012] According to another aspect of this disclosure, an edge-cloud collaborative incremental optical character recognition method is provided, applied to a scanning pen device, comprising: Local stitching is performed on partially overlapping images acquired in successive acquisitions to form a local stitching canvas; Starting from the uploaded width, and taking the difference between the current effective width of the local splicing canvas and the uploaded width as the strip width, export the newly added column image from the local splicing canvas as the image strip; Semantic segments are generated and saved in a history cache indexed by session sequence number. The semantic segments carry session sequence number. The semantic segments include keyframe segments generated at the beginning of local splicing segmentation, carrying the starting segment image of the local splicing canvas, and strip segments carrying the image strips and tail append markers. The tail append markers are used to indicate that the image strips have completed local spatial alignment and should be appended to the tail boundary of the effective area of ​​the incremental canvas maintained by the server. The semantic fragment is sent to the server, and after successful transmission, the uploaded width is updated to the current valid width. When local splicing fails and the preset rollback conditions are met, or when the local splicing canvas reaches its capacity limit, a rollback request carrying a checkpoint sequence number is sent to the server, and the uploaded width is cleared to zero and the local splicing segment is reset. The checkpoint sequence number indicates the session sequence number of the last trusted semantic segment. Based on the retransmission start sequence number returned by the server, retransmit semantic fragments with in-session sequence numbers not less than the retransmission start sequence number from the historical cache, or send recalculated replacement semantic fragments, and send keyframe fragments for re-anchoring the incremental canvas to the server for the reset local splicing segment.

[0013] In a preferred embodiment, the method further includes determining whether the local splicing failure meets the preset rollback condition in the following manner: When the absolute value of the horizontal displacement estimated by local splicing is less than the displacement dead zone, the horizontal displacement is set to zero and the zero displacement count is accumulated. When the zero displacement count reaches a predetermined number, it is determined that a splicing failure has occurred. For stitching failures determined to be caused by low texture or momentary lack of motion, only the anchor frames used for local stitching are updated without sending the rollback request; For any remaining stitching failures, first re-execute the local stitching once for the next local image. If the re-execution still fails, send the rollback request.

[0014] According to another aspect of this disclosure, an electronic device is provided, including a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the methods as described above.

[0015] In the embodiments of this disclosure, the server receives keyframe fragments carrying session sequence numbers and strip fragments carrying aligned image stripes and tail appending marks. When the sequence number equals the expected sequence number, the server accepts the fragments and writes them into a replayable semantic fragment ledger. The server initializes the incremental canvas based on the keyframe fragments and appends the image stripes directly to the tail boundary of the effective area based on the tail appending marks. When the recognition trigger condition is met, the server performs phased optical character recognition and updates the recognition trigger baseline. In response to a rollback request carrying a checkpoint sequence number, the server truncates the ledger, clears the incremental canvas and derived phased recognition results and recognition trigger baseline, replays the preserved fragments, reconstructs the canvas, and returns the retransmission starting sequence number. This allows the server to avoid repeating the process. Full-frame motion estimation can quickly construct a recognizable canvas based on the newly added columns aligned on the device. It provides stage recognition results during the scanning process, thereby shortening the waiting time for the first usable text. The uploaded content corresponds only to the columns that actually grow in the local spliced ​​canvas, thus avoiding the proportional increase of duplicate pixels with the number of captured frames. When the device determines that the state after the checkpoint is unreliable, it simultaneously cancels the impact of the erroneous stripe on the canvas pixels, recognition results, and recognition scheduling state, restoring to a reliable checkpoint at a cost less than a full retransmission. This avoids the end-to-cloud state fork where the device has already corrected the error but the server still uses the erroneous canvas and old results. It also avoids the reconstructed canvas not generating stage recognition results for a long time because it uses the old trigger baseline.

[0016] Furthermore, by enabling semantic fragments to carry canvas generations, only fragments whose canvas generations match the current canvas generation are accepted, and a new canvas generation is established and returned to the device with the retransmission start sequence number after the rollback and replay are completed, fragments in the old generation that are still being transmitted in the network before the rollback and whose sequence number is exactly the same as the expected sequence number of the new generation can be rejected, thus avoiding the reuse of sequence numbers that could lead to the old fragments mistakenly modifying the incremental canvas.

[0017] Furthermore, by enabling semantic fragments to carry the preceding state fingerprint, and by chaining the preceding prefix state fingerprint, the current canvas generation, sequence number, fragment type, placement semantics, and image payload summary to calculate the prefix state fingerprint, fragments are applied only when the preceding state fingerprint matches the current prefix state fingerprint, and the checkpoint prefix state fingerprint is returned as the baseline fingerprint for the new generation after rollback, a verifiable correspondence can be established between the incremental canvas state and the ledger event prefix, fragments with consecutive sequence numbers but from another state branch are rejected, thus avoiding state branch confusion.

[0018] Furthermore, by enabling the stage recognition results to carry a version containing the canvas generation, the applied sequence number, and the prefix state fingerprint, and only outputting the recognition results whose canvas generation is consistent with the current canvas generation and whose prefix state fingerprint belongs to the current ledger fingerprint sequence, old asynchronous recognition results that are returned late after rollback commit can be discarded, thus avoiding them from polluting the recognition output corresponding to the current canvas.

[0019] Furthermore, by using the execution time and content amount of the previous stage recognition as the recognition trigger baseline, and requiring that stage recognition be triggered only when the time difference reaches the time threshold and the content growth reaches the content growth threshold, it can avoid repeatedly recognizing the same canvas when the scanning pen is stationary simply because the time has arrived, and also avoid calling recognition for each small stripe, so that the recognition call volume is related to the newly added recognizable content, and the trigger baseline is re-established based on the reconstructed canvas after rollback.

[0020] Furthermore, by prioritizing the use of the complete mosaic image uploaded by the device at the end of the scan, and using the image within the effective area of ​​the incremental canvas as the final state recognition input when the complete mosaic image is not received, it is possible to provide interim results without waiting for the complete mosaic image during the scan process, obtain the final state recognition result using the higher quality complete mosaic image at the end of the scan, and still complete the final state recognition even when the complete mosaic image is missing.

[0021] Furthermore, by appending the keyframe fragments received after the effective region initialization as re-anchor frames to the tail boundary, and by including the semantic fragments as strip fragments to be located that carry placement parameters but not tail appending markers, and by performing a legality reduction on the placement parameters of the strip fragments to be located before applying them, the keyframes resent by the device after segmentation or rollback due to capacity limits can form a continuous canvas with the existing content on the server, maintaining the text order across segments. The abnormal or out-of-bounds placement parameters reported by the device are also reduced to legal values ​​without destroying the existing canvas content, making the incremental canvas robust to abnormal placement parameters from the device.

[0022] Furthermore, by exporting image strips from the local splicing canvas on the device side, starting with the uploaded width and using the difference between the current effective width and the uploaded width as the width, keyframe fragments are generated at the beginning of the local splicing segment, and strip fragments carrying session sequence numbers and appended markers are generated and stored in the history cache indexed by sequence number. After successful transmission, the uploaded width is updated. When splicing fails and the preset rollback conditions are met or the local splicing canvas reaches its capacity limit, a rollback request carrying a checkpoint sequence number is sent and the local splicing segment is reset. Based on the retransmission start sequence number, keyframe fragments are sent from the history cache or replaced semantic fragments are sent, and the keyframe fragments are sent for the reset segment. This ensures that the uploaded bytes correspond only to the columns that actually grow in the local splicing canvas and eliminates the need for secondary registration on the server side. After rollback, the image before the checkpoint can be accurately resumed without re-uploading, and the new segments after segment rotation or rollback reset are continuously anchored on the server side.

[0023] Furthermore, by setting the horizontal displacement with an absolute value less than the displacement dead zone to zero and only determining stitching failure when the zero displacement count reaches a predetermined number, stitching failures caused by low texture or momentary lack of movement are only updated with anchor frames without rollback. For other stitching failures, a local retry is performed first, and a rollback request is only sent if the retry still fails. This can suppress the accumulation of pseudo-displacement when the scanning pen is stationary or slightly jittery, so that the costly rollback only occurs when the local retry is ineffective, and avoids repeated reconstruction of the end cloud state due to momentary low texture during normal scanning.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Various aspects and embodiments of this disclosure can be combined in different ways. This part of the disclosure is not intended to identify key or essential features of the claimed subject matter, nor is it intended to aid in determining the scope of the claimed subject matter. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of an edge-cloud collaborative incremental optical character recognition system according to an embodiment of the present disclosure; Figure 2 This is a flowchart of an end-to-cloud collaborative incremental optical character recognition method applied to a server according to an embodiment of the present disclosure; Figure 3 This is a flowchart of an end-to-cloud collaborative incremental optical character recognition method applied to a scanning pen device according to an embodiment of the present disclosure.

[0026] Figure labeling: 100 is the scanning pen device; 200 is the communication network; 300 is the incremental adaptation service; 310 is the session and sequence verification module; 320 is the semantic fragmentation ledger; 330 is the fast canvas module; 340 is the stage optical character recognition scheduling module; 350 is the rollback module; 360 is the final state input selection module; 400 is the upstream optical character recognition engine. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the disclosure will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1 This embodiment provides an edge-cloud collaborative incremental optical character recognition method applied to the server side. The server can be a public cloud server, an enterprise intranet server, an edge gateway on the same local area network as the scanner, or a computing-capable device such as a mobile phone connected to the scanner. The upstream optical character recognition engine can be deployed on the same machine as the server or invoked via a remote interface. The scanner continuously acquires partially overlapping local images at a high frame rate and stitches them together locally. The server only receives keyframes and new content uploaded by the device, continuously maintaining an incremental canvas for recognition during the scanning process. The overall process of this method is as follows: Figure 2 As shown, the process mainly includes six stages: receiving semantic fragments, accepting and recording according to the sequence number in the session, initializing the incremental canvas, appending image stripes to the end, optical character recognition during the execution stage, and responding to rollback requests. Each stage will be explained in detail below.

[0029] Step 101: Receive semantic segments from the scanning pen device. A semantic segment is an information unit that binds an image payload with application-layer placement semantics, and its content can change the image state on the server. In this embodiment, each semantic segment carries at least a session identifier, a session sequence number, a segment type, and an image payload. The session sequence number increments continuously from a pre-defined starting value within a scanning session, uniquely determining the application order of the segment within the session. Based on segment type, semantic segments include at least keyframe segments and strip segments. Keyframe segments carry a complete image segment of the locally stitched canvas on the device, used to initialize the incremental canvas on the server; strip segments carry image strips and end-append markers. In some implementations, strip segments may not carry end-append markers but instead carry placement parameters such as the added width and vertical displacement. The server determines the pasting position based on these placement parameters. These strip segments are referred to as unpositioned strip segments in this document, and their processing method will be described later. Image stripes are new column images exported by the scanning pen device from its local mosaic canvas; specifically, the device records the width of the previously uploaded image. When the local mosaic canvas grows to the current effective width At that time, the device directly exports the column range from the local splicing canvas. All pixel columns within are used as image strips, where This is the width of the uploaded file. The current effective width, both are measured in pixels (columns), and the width of the image strip is... Since the image stripes are already located in the coordinate system of the local splicing canvas during export, meaning spatial alignment has been completed on the device side, the device side sets a tail append marker in the strip fragments to indicate that the server does not need to perform displacement search or vertical registration on the stripes, but should directly append them to the tail boundary of the effective area of ​​the incremental canvas. The tail append marker can be represented by an independent Boolean field, or by a predetermined flag value or enumeration value of a field in the fragment, such as setting the new width field to a negative value as the tail append sentinel value. This disclosure does not limit the specific field form. The image payload can use encoding methods such as grayscale JPEG, PNG, or 8-bit raw grayscale data. The transport layer can use LZ4, Zstandard, zlib compression, or no compression. For raw grayscale data, semantic fragments simultaneously carry width, height, and row span so that the server can recover the two-dimensional image. This embodiment takes 8-bit raw grayscale data, a row span equal to the strip width, and first-level zlib compression as an example. Semantic fragments can be uploaded one by one by the device, or multiple consecutive semantic fragments can be uploaded in batches by the device, and the server processes them one by one in array order.

[0030] Step 102: When the in-session sequence number equals the expected sequence number, accept the semantic fragment. Write the accepted semantic fragment into the replayable semantic fragment ledger according to the in-session sequence number, and advance the expected sequence number. The server maintains an expected sequence number for each scanning session, with its initial value consistent with the starting sequence number agreed upon by the device. Upon receiving a semantic fragment, the server compares its in-session sequence number with the expected sequence number: if they are equal, accept the fragment, append the fragment's in-session sequence number, fragment type, frame identifier, placement semantics, and decoded grayscale image as a record to the semantic fragment ledger, and then increment the expected sequence number. As a further processing step, when the in-session sequence number is less than the expected sequence number, it indicates that the fragment is a duplicate fragment caused by network retransmission. The server returns an acknowledgment to the device but does not apply it again to the incremental canvas, thus ensuring that the application process is idempotent. When the in-session sequence number is greater than the expected sequence number, it indicates that there is a fragment that has not yet arrived. The server refuses to process the fragment and returns the current expected sequence number to the device as the retransmission starting sequence number. In batch uploads, the server verifies and applies data sequentially, stopping at the first failed fragment and returning the number of received fragments, the error index, and the sequence number requiring retransmission. This prevents fragments following the failed fragment from being skipped during execution. The semantic fragment ledger is a truncated sequence of image events stored in session-in-session order, which can be reused in its original order. It can store both decoded grayscale images and the original compressed payload, which can be re-decoded during replay. The ledger is not only used for auditing but also serves as the event source for subsequent rollback and reconstruction. The receive, rollback, and termination operations within the same session are serialized on the server by the session's reentrant lock, allowing concurrent processing between different sessions.

[0031] Step 103: Initialize the incremental canvas and its effective area based on the keyframe fragments. When the server receives the first keyframe fragment in the session, it creates a grayscale incremental canvas, places the keyframe image in the vertical center of the canvas, and initializes the effective area. The effective area refers to the region in the incremental canvas that has been occupied by the effective image content, which can be represented by the used bounding box and cursor. Its right boundary is the tail boundary of the effective area. To reduce the number of subsequent dynamic canvas expansions, this embodiment uses an initial canvas height of 3 times the keyframe height and an initial canvas width of 8 times the keyframe width as an example. This ratio is an exemplary design parameter and can be adjusted according to the device's canvas size and the server's memory resources. After initialization, the tail boundary of the effective area is located at the right edge of the keyframe image, and the cursor is located at this edge.

[0032] Step 104: Based on the tail appending marker, append the image strip to the tail boundary of the effective area of ​​the incremental canvas, and update the effective area according to the width of the image strip. For strip segments carrying tail appending markers, the server no longer performs a displacement search on the entire canvas, but directly uses the current tail boundary of the effective area as the starting x-coordinate for pasting the strip, and the current cursor y-coordinate as the pasting y-coordinate. Let the x-coordinate of the current tail boundary of the effective area be... The current cursor's ordinate is The pasting position of the top left corner of the image strip in the incremental canvas. satisfy and ,in This is the x-coordinate of the paste location. The vertical coordinate of the paste position. The x-coordinate of the effective region's tail boundary. This represents the ordinate of the current cursor, in pixels. After pasting, since the width of the image stripe is exactly equal to the width added to the local splicing canvas on the device in this round, the server updates the tail boundary of the valid area with this stripe width. The server dynamically expands the canvas to the right or up and down, and simultaneously updates the cursor and the used bounding box. When the pasting target position exceeds the current canvas capacity, the server dynamically expands the canvas to the right or up and down, and simultaneously translates the cursor and the used bounding box. For the strip segments to be positioned based on the placement parameters as described later, when there is a small overlap between the pasting area and existing content, dark value priority merging is performed on the overlapping area to preserve character strokes, and the non-overlapping area is pasted directly. For strip segments with tail appending marks, the pasting starting point is the tail boundary, and there is no overlap with existing content. Thus, without repeatedly performing high-cost full-frame motion estimation, the server can continuously build an incremental canvas corresponding to the local splicing canvas on the device, based solely on the placement semantics provided by the device.

[0033] Step 105: When the recognition trigger condition is met, perform intermediate optical character recognition (OCR) on the image within the valid area of ​​the incremental canvas and update the recognition trigger baseline. Intermediate OCR refers to intermediate recognition performed on the image within the current valid area before the scan is finished. Its purpose is to provide the user with usable text results during the scanning process. The server maintains a recognition trigger baseline for each session. The recognition trigger baseline is the benchmark used to determine whether the recognition trigger condition is met. For example, it could be the execution time of the previous intermediate OCR, the content of the valid area at the time of this recognition, or a combination of both. Whenever a semantic segment is successfully applied to the incremental canvas, the server compares the current state of the incremental canvas with the recognition trigger baseline to determine if the recognition trigger condition is met. If it is, the image within the valid area is cropped from the incremental canvas using the used bounding box, and pure white edges are further removed. The cropping result is submitted to the upstream OCR engine for intermediate OCR, and then the recognition trigger baseline is updated to the state at the time of this recognition, such as the execution time of this recognition and the content of the valid area at the time of this recognition. When a valid canvas is first encountered in the session, the server can perform a staged optical character recognition (OCR) without being limited by the historical baseline. It can also force a staged OCR at the end of the scan. After recognition, the server records the recognized text, average confidence level, update time, number of recognition attempts, canvas width during recognition, and time consumption. The server then returns the staged recognition results to the device via bounded long polling, message push, or other asynchronous methods.

[0034] Step 106: In response to a rollback request carrying a checkpoint sequence number from the scanning pen device, perform ledger truncation, state clearing, deterministic replay, and return the retransmission start sequence number. In cases of local stitching failure or local stitching segment reset, the device will send a rollback request to the server. The rollback request must at least carry the checkpoint sequence number. Checkpoint number This represents the session sequence number of the semantic shard that the device considers the last trusted one. The rollback request can also carry a triggering reason. After receiving the rollback request, the server atomically performs the following processing within the lock of that session: First, it deletes all sessions with sequence numbers greater than [a certain value] from the semantic shard ledger. Semantic fragmentation, and set the expected sequence number to Then, clear all states derived from the deleted semantic fragments, including the current incremental canvas and its cursor and used bounding boxes, the stage optical character recognition results generated based on the deleted fragments, and the recognition trigger baseline, such as the execution time of the last recognition, the amount of content at the time of the last recognition, and the number of stage recognitions; in addition, the final state recognition results, error states, and references to the complete mosaic image can also be cleared. Next, in ascending order of session sequence number, the keyframe fragments and strip fragments retained in the semantic fragment ledger are reapplied to a new incremental canvas, that is, these fragments are replayed in the manner of steps 103 and 104, thereby reconstructing the incremental canvas to an image state consistent with the checkpoint. Finally, the retransmission start number determined based on the checkpoint sequence number is returned to the device. In this embodiment, the retransmission start number is [insert retransmission start number here]. The device then retransmits semantic fragments starting from the retransmission start number. This replay is deterministic because each record in the ledger stores the complete placement semantics and image payload. Reapplying them in the original order will inevitably yield the same canvas content as at the checkpoint before the rollback, so the device does not need to re-upload any images before the checkpoint. Clearing the recognition trigger baseline is particularly important. If the trigger baseline of an incorrect canvas is used, the reconstructed effective area may not trigger staged optical character recognition for a long time because the content volume or the elapsed time has not increased sufficiently relative to the old baseline. After clearing, the reconstructed canvas and its subsequently added strips can again generate effective staged recognition results according to the recognition trigger conditions. Therefore, the object of rollback is not the network byte stream itself, but rather the semantic events that can change the canvas and their derived recognition states.

[0035] Steps 101 to 106 above constitute a complete server-side processing solution. Those skilled in the art can implement incremental canvas construction, stage identification, and checkpoint rollback on the server side based on the above steps. Its working principle is as follows: the device side only uploads keyframes and aligned new columns. The server side treats each fragment as a replayable application layer event, records it sequentially, and directly appends it to the end, so that the incremental canvas, stage identification results, and identification trigger baseline are all derived from the same event sequence. Once the device side determines that the state after a certain checkpoint is unreliable, the server side can simultaneously undo the impact of the erroneous stripe on the canvas pixels and on the identification results and scheduling state by truncating the ledger and replaying the prefix, thus avoiding the end-cloud state fork where the device side has corrected the error but the server side still displays the old results. Compared to uploading the complete image all at once after scanning, this method can generate stage recognition results during the scanning process, which helps to shorten the waiting time for the first usable text; compared to uploading the original image frame by frame, the transmitted object is mainly the newly added column, and the repeated pixels do not increase proportionally with the number of captured frames; compared to general sequence number retransmission, deterministic replay restores the integrity of the image processing state machine, not just the network payload. Several optional improvements to this embodiment are described below.

[0036] In some optional implementations, the recognition trigger baseline in step 105 specifically includes the execution time of the previous stage of optical character recognition. and the amount of content in the effective region during this recognition. The trigger condition for identification is set to be met simultaneously by both a time threshold and a content growth threshold, i.e. in The current time is the time displayed on the server's monotonic clock. This refers to the execution time of the previous stage of optical character recognition. As a time threshold, The current content volume of the valid region. To determine the amount of content in the effective region during the previous stage of optical character recognition, The content growth threshold is set in milliseconds, and the content volume in this embodiment is measured in the width of the effective area, i.e., the number of pixel columns. The content volume can also be measured by other metrics such as the area of ​​newly added foreground pixels, the area of ​​the effective bounding box, the number of text detection boxes, or the cumulative bytes of newly added stripes. In this embodiment, the time threshold and content growth threshold can be exemplarily set to 1000 milliseconds and 12 pixels respectively. Both can be adaptively adjusted based on network conditions, recognition engine load, font size, or user scanning speed, or can be obtained from fixed values, lookup table values, or dynamically calculated based on network round-trip latency and upstream recognition queue length. The synergistic effect of the dual thresholds is that the time threshold limits high-frequency calls, and the content growth threshold avoids repeated recognition of the same or nearly identical images. Both simultaneously ensure that the upstream recognition call volume is related to the newly added identifiable content rather than simply the number of frames; when the scanning pen is stationary, it will not repeatedly recognize the same canvas simply because time has elapsed, and when moving rapidly, it will not call recognition for every small strip. After the rollback, since step 106 has been cleared... and The dual threshold will re-establish the baseline based on the reconstructed effective area. It should be noted that the dual threshold is not a necessary condition for implementing this method. When only a single time threshold or a single content growth threshold is used as the identification trigger condition, the accounting, appending and rollback processes of this method can still operate normally.

[0037] Building upon the above scheme, the final state optical character recognition (OCR) input selection can be further configured upon completion of the scan. Scan completion can be triggered by an explicit end request from the device or by a complete frame fragment with a termination marker. The complete frame fragment carries the complete stitched image generated by the device after scan completion. The device can perform post-processing on the complete stitched image locally, including content cropping, initial background cleanup, top dashed line artifact suppression, and text edge enhancement. In response to scan completion, if the server has received the complete stitched image, it uses the cropped image as the input for final state OCR; if it has not received the complete stitched image, it crops the image within the valid area from the used bounding box of the incremental canvas and removes the pure white edges, using this as the input for final state OCR; if neither exists, the session is set to an error state. During the scanning process, the server does not wait for the complete mosaic image but provides interim results using an incremental canvas. At the end of the scan, the final input is replaced with the complete mosaic image, thus decoupling the low-latency path from the high-quality path. Even if the complete mosaic image fails to arrive due to network issues, the incremental canvas can still provide a usable, fault-tolerant final input. Optionally, the final input can be selected between the complete mosaic image and the incremental canvas based on image sharpness, number of seams, recognition confidence, or integrity score, or the results can be merged after performing recognition on both separately, based on character confidence, longest common subsequence, or language model. When not using the above input selection, either the incremental canvas or the complete mosaic image can always be used as the input for final recognition; the remaining steps of this method remain unaffected.

[0038] As a preferred but not mandatory improvement, the server can differentiate keyframe fragments received within the same session after the effective area of ​​the incremental canvas has been initialized: these keyframe fragments are treated as re-anchored frames, and the server appends them to the tail boundary of the effective area, rather than creating a new canvas or overwriting existing content. When the device performs segmentation rotation due to reaching the capacity limit of the local spliced ​​canvas, or when the uploaded width is cleared to zero and keyframes are resent due to rollback, the server can still splice multiple local segments into a continuous effective canvas, and the text order across segments is maintained. At the same time, the server can perform legality reduction on the placement parameters carried by semantic fragments before applying them. Legality reduction includes taking non-negative values ​​for the added width and clamping the vertical displacement within a predetermined range. This embodiment takes a vertical clamping range of ±24 pixels as an example, and verifies the relationship between the target area and the canvas capacity before application. It should be noted that in the implementation using a negative value of the "Add Width" field as the sentinel value for the tail append marker, the server first determines whether the slice is a tail appender based on this field. For slices determined to be tail appenders, it directly processes them according to step 104 without executing the non-negative value restriction. Only for slices to be located that do not carry a tail append marker, the following legality restriction is executed. For slices carrying an "Add Width" field... With longitudinal displacement For strip segments to be positioned that are not marked at the tail, their pasting position can be determined according to... Confirmed, among which , Paste the x and y coordinates to the top left corner of the strip. , These are the x and y coordinates of the current cursor. The increased width reported by the device. This refers to the longitudinal displacement reported by the device. This represents the upper limit of the vertical clamping range. Indicates taking zero and The larger one, Indicates will Restricted to closed intervals The clamping functions within the function are all in pixels. Abnormal or out-of-bounds placement parameters are thus normalized to valid values ​​instead of destroying existing canvas content, making the incremental canvas robust to abnormal output from the device. It should be noted that the re-anchoring frame processing and the validity regulation are independent and can be used alone; without re-anchoring frame processing, a new session can be created at each segment rotation. Without the validity regulation, this method can also append strip segments with trailing append markers normally.

[0039] Example 2 This embodiment, based on Embodiment 1, further introduces canvas generation, prefix state fingerprint, and recognition result version to address three types of concurrency issues that may arise when relying solely on in-session sequence number rollback: sequence number reuse after rollback leading to the erroneous acceptance of old in-transit fragments; branch confusion due to identical sequence numbers but different state prefixes; and result contamination caused by the delayed return of asynchronous recognition results after rollback commit. The features described below are all optional improvements to Embodiment 1, and the three are progressively enhanced, with the latter building upon the former.

[0040] In some alternative implementations, the server maintains a canvas generation for each session. Canvas generations are used to identify a consistent state lineage within a session. Take a non-negative integer. At the start of the session, an initial generation is established as the current canvas generation. Semantic fragments carry the canvas generation in addition to the session sequence number. The acceptance condition in step 102 is correspondingly expanded to: a semantic fragment is accepted only if the canvas generation carried by the semantic fragment is equal to the current canvas generation and the session sequence number is equal to the expected sequence number; semantic fragments carrying a canvas generation different from the current canvas generation are rejected. After replay is completed in step 106, the server atomically establishes a new canvas generation different from the canvas generation before rollback as the current canvas generation. For example, let... The new canvas generation is then sent back to the device along with the retransmission start sequence number. Therefore, even if there are still fragments in the old generation still being transmitted in the network, and their in-session sequence numbers happen to match the expected sequence numbers of the new generation, they will be rejected due to the canvas generation mismatch, and the incremental canvas will not be changed again. The canvas generation can also be generated in other situations where the server determines that a state hierarchy switch is needed, such as during re-anchoring. It should be noted that the canvas generation is not a necessary condition for implementing this method. Without setting a canvas generation, the server can reduce the impact of sequence number reuse by requiring the device to delay retransmission after rollback, or by ensuring at the transport layer that all old fragments have arrived or been discarded.

[0041] As a preferred but not mandatory improvement, in addition to setting canvas generations, semantic fragments can also carry predecessor state fingerprints, while the server maintains a chain-computed prefix state fingerprint. Specifically, the server receives each session with a sequence number of... The semantic fragment is then used to calculate a new prefix state fingerprint based on the previous prefix state fingerprint, the current canvas generation, the session sequence number of the semantic fragment, the fragment type, the placement semantics, and the image payload summary. in To accept the first The prefix state fingerprint after each semantic segmentation. To accept the first The prefix state fingerprint after each semantic segment is the same as the previous prefix state fingerprint, used by the first segment in the session. The fingerprint is either the agreed-upon initial fingerprint or the baseline fingerprint for the current generation. For the current generation of canvases, This is the in-session sequence number of the semantic fragment. This refers to the sharding type of this semantic sharding. The placement semantics for this semantic segment include appending markers to the tail or placing parameters for image stripes, such as adding width and vertical displacement. The image payload for this semantic segment. This is a function that calculates a summary of the image payload; its output is the image payload summary. This represents a byte string concatenation operation. The hash algorithm can be SHA-256, BLAKE3, a combination of cyclic redundancy check and session key, or other hash algorithms that meet collision detection requirements. The input to the prefix state fingerprint can further include at least one of the following: effective canvas boundary, stripe width, encoding type, and compressed payload digest. When generating each semantic fragment, the device writes the server's current prefix state fingerprint, which it knows, as the predecessor state fingerprint into the fragment. The server applies the semantic fragment to the incremental canvas only if the predecessor state fingerprint carried by the semantic fragment is equal to the current prefix state fingerprint. That is, accepting and applying a fragment requires simultaneously satisfying three conditions: canvas generational matching, session sequence number equal to the expected sequence number, and predecessor state fingerprint equal to the current prefix state fingerprint. Therefore, fragments with consecutive session sequence numbers but originating from another state branch cannot be mistakenly received, and a verifiable correspondence is established between the current state of the incremental canvas and the event prefix of the semantic fragment ledger. After replaying in step 106, the server recalculates the fingerprint for each of the retained fragments according to the above formula to obtain the checkpoint prefix state fingerprint at the checkpoint. and will As a benchmark fingerprint for the new generation of canvas The new canvas generation and retransmission start number are returned to the device along with the new canvas generation and retransmission start number. For checkpoint number, This serves as the benchmark fingerprint for the next generation. The device uses this benchmark fingerprint. As the preceding state fingerprint of the first subsequent fragment, historical fragments are replayed or corrected replacement fragments are sent, thus forming a new generation of fingerprint chains for each fragment. During batch uploads, the server stops at the fragment where the first canvas generation, session sequence number, or preceding state fingerprint does not match, and returns the error index, current canvas generation, expected sequence number, and current prefix state fingerprint. When prefix state fingerprints are not used, fragments in transit from older generations can be rejected based solely on the canvas generation and session sequence number. Prefix state fingerprints provide further protection against state branch confusion.

[0042] To further eliminate delayed pollution after asynchronous recognition calls are rolled back, in addition to setting a prefix state fingerprint, the recognition results of staged optical character recognition can also carry a result version. Result Version Including the canvas generation at the time of this identification, the applied in-session sequence number, and the prefix state fingerprint, it can be represented as: ,in To determine the current canvas generation at the time of this recognition, The session sequence number of the last semantic segment applied in the ledger at the time of this identification. This is the prefix state fingerprint at this point; the result version may further include the boundary of the valid region during recognition or the recognition request identifier. The server performs version verification when the recognition result is returned. The recognition result is only output if the canvas generation in the result version equals the current canvas generation, and the prefix state fingerprint in the result version belongs to the current prefix state fingerprint sequence of the semantic shard ledger. Recognition results that do not meet the aforementioned conditions are discarded. Here, the prefix state fingerprint sequence refers to the sequence calculated sequentially according to the shards retained in the current ledger. The sequence in which the fingerprint in the resulting version belongs to that sequence means that the canvas state used for recognition is a prefix of the current state spectrum, and the recognition result is still valid. Version verification can be performed on the server-side result gateway or by the device before display. Old asynchronous recognition results returned after the rollback transaction is committed have an old canvas generation or their fingerprints are not in the fingerprint sequence of the new ledger, and are therefore discarded and not displayed to the user. Optionally, the server can first try to cancel the ongoing recognition request during rollback, and if it cannot be canceled, then perform the above version filtering on the return path. The results of final-state optical character recognition can also carry the result version and be filtered in the same way. It should be noted that without using the result version, the server can simply clear the cached stage recognition results during rollback, and the rest of this method will not be affected. The result version is a further improvement on extending the server-side cache clearing to end-to-end result validity determination.

[0043] In summary, the canvas generation constrains input events, the prefix state fingerprint constrains state evolution, and the result version constrains output results. These three elements maintain end-to-end consistency under three concurrency conditions: old generation fragments in transit, state branch confusion, and delayed asynchronous recognition results. This forms an end-to-end consistency closed loop from image input events and canvas state evolution to recognition result output. The ledger truncation, canvas reconstruction, recognition result invalidation, recognition trigger baseline reset, new canvas generation establishment, and rollback response are all completed within the same session's lock, forming atomic state transitions.

[0044] Example 3 This embodiment provides an edge-cloud collaborative incremental optical character recognition method, applied to a scanning pen device, and working in conjunction with the server-side method described in Embodiment 1 or Embodiment 2. The device can be the processing unit of the scanning pen itself, or a processing unit integrated with the scanning pen, responsible for image acquisition, local stitching, incremental column export, and protocol transmission. The overall flow of this method is as follows: Figure 3 As shown, the process mainly includes six steps: local splicing, exporting image strips, generating and caching semantic fragments, sending semantic fragments and updating the uploaded width, sending rollback requests, and resuming transmission based on the retransmission start sequence number.

[0045] Step 201: Locally stitch together partially overlapping images acquired in consecutive captures to form a local stitching canvas. In one specific implementation, the scanning pen uses a long-press scanning button semantic; pressing the button starts the scanning session and enables fill light and frame callback, while releasing the button ends the scanning session. The device acquires partial images through a camera callback with a pre-allocated buffer. After completing grayscale processing in the callback thread, the buffer is returned to the camera as soon as possible. The stitching thread samples from the double buffer at preset time intervals, decoupling the frame rates of acquisition, display, and stitching. In this embodiment, the acquired image is cropped into a fixed region of interest and downsampled to a grayscale frame with a width of 168 pixels and a height of 120 pixels, and the local stitching canvas height is 376 pixels. The above dimensions are only example parameters; any overlapping text content between adjacent frames is acceptable. For adjacent grayscale frames, the device estimates the horizontal and vertical displacements between them, and then stitches the non-overlapping parts of the current frame to the end of the local stitching canvas according to the estimated displacements. Displacement estimation can employ features such as row and column projection matching, normalized cross-correlation, template matching, phase correlation, optical flow, FAST, or ORB, or it can combine the output of inertial sensors or optical navigation sensors; this disclosure does not limit this. This embodiment takes a multi-candidate motion estimation as an example: first, a coarse displacement is obtained from the difference between the column and row projections of adjacent frames; then, relevant candidates are obtained within the neighborhood of the coarse displacement using normalized cross-correlation; and prior motion candidates are predicted based on the exponential moving average of historical velocities and the current frame interval. Subsequently, a fine search is performed on each candidate within a finite neighborhood using dark pixel-weighted grayscale difference, ensuring that the contribution of character strokes to displacement evaluation is higher than that of the paper background. Finally, the displacement with the minimum total cost is selected. To avoid immediately outputting stripes after each sampling frame, the device accumulates the effective horizontal displacement and weights the vertical displacement accordingly. Once a threshold is reached, a new width and vertical displacement are generated. This embodiment uses a cumulative threshold of 6 pixels and a tail refresh threshold of 2 pixels at the end of the scan as an example. At the seams between the new content and the existing canvas, overlapping positions with similar dark pixels of characters are assigned darker values ​​to preserve strokes, while a smoothing weight is applied to the bright background. The motion estimation and seam processing described above are examples implemented internally by the device, and the output is a locally spliced ​​canvas that continuously grows to the right.

[0046] Step 202: Starting from the uploaded width, and using the difference between the current valid width of the local mosaic canvas and the uploaded width as the strip width, export the newly added column image as the image strip from the local mosaic canvas. The device maintains the uploaded width. This indicates the number of columns in the local tiled canvas that have been successfully uploaded to the server. Its initial value is zero or the width of the first keyframe; the current effective width of the local tiled canvas. This indicates the number of columns in the canvas that are currently occupied by valid content. When Greater than At that time, the device exports column ranges row by row through the native interface of the local splicing canvas. Within the pixels, the width is obtained. Image stripes with a height equal to the height of the local mosaic canvas, where , The meaning is the same as before. Since the image stripes are directly taken from the local splicing canvas, each column has already been spatially aligned with the preceding content locally. The export process does not require any further displacement calculations, and the uploaded bytes only correspond to the columns that have actually grown in the local splicing canvas. To limit the generation frequency of the stripes, the device can also set a minimum upload interval for the stripes. In this embodiment, 300 milliseconds is used as an example, and export is only performed when this interval has been reached since the last export and the canvas has indeed grown.

[0047] Step 203: Generate semantic segments carrying session sequence numbers, image stripes, and tail appending markers, and save the generated semantic segments in the historical cache indexed by session sequence numbers. The device assigns a continuously increasing session sequence number to each semantic segment within the session, uses the image strip as the image payload, and sets a tail appending marker to indicate that the image strip has completed local spatial alignment. The server should append it to the tail boundary of the effective area of ​​the incremental canvas maintained by the server. Semantic segments also carry fields such as session identifier, segment type, frame identifier, encoding method, compression method, width, height, and line span. The image payload can use 8-bit raw grayscale data compressed with zlib, or it can use grayscale JPEG, PNG, or other encoding methods. Semantic segments can also carry a device-side monotonic timestamp for the server to calculate end-to-end latency. At the start of the session or at the start of each local stitching segment, the device also generates a keyframe segment. The keyframe segment carries the starting segment image of the local stitching canvas and is used by the server to initialize the incremental canvas. Each generated semantic fragment is simultaneously stored in a historical cache indexed by its session sequence number. The historical cache is a bounded cache with a fixed capacity, which evicts the earliest fragments by their session sequence number. Its capacity should be no less than the number of fragments covered by the rollback described later, so that it can be replayed according to the retransmission start sequence number returned by the server. The historical cache can also store the payload summary and predecessor state fingerprint of each fragment, so that it can be replayed as is or replaced and retransmitted in the fingerprint chain scheme of Embodiment 2.

[0048] Step 204: Send the semantic fragment to the server and update the uploaded width to the current valid width after successful transmission. The device can send semantic fragments one by one, or it can put the semantic fragments to be sent into a sending queue and send them in batches when the number of fragments in the queue reaches the batch number threshold or the refresh time threshold since the last transmission. Semantic fragments can be carried by HTTP, HTTP / 2, WebSocket, QUIC, message queues, or custom reliable transport protocols. Successful transmission means the server returns an acknowledgment that it has accepted the fragment; only on paths where transmission is successful will the device update the uploaded width. Updated to the current valid width when exporting this strip. This ensures that the next exported image stripe starts exactly from the right edge of the previous stripe, without omitting or repeating any columns. If the server returns a response indicating an abnormal sequence number and carrying a retransmission start number, the device will retransmit from the historical cache as described in step 206.

[0049] Step 205: When local stitching fails and meets preset rollback conditions, or when the local stitching canvas reaches its capacity limit, a rollback request carrying a checkpoint sequence number is sent to the server. There are two types of situations that trigger a rollback request. One is capacity-based segment rotation, where the local stitching canvas reaches the preset maximum width, requiring the device to end the current local stitching segment and create a new one. This situation is expected and does not necessarily indicate a stitching error. The other is mismatch-based rollback, where local stitching fails and meets preset rollback conditions, such as consecutive zero displacement, matching failure, abnormal scanning direction, canvas exceeding limits, failure after retries, or the device actively resetting the current stitching segment. Both situations cause the device to send a rollback request to the server, and the rollback request must carry at least a checkpoint sequence number. It can also carry a trigger reason. The checkpoint sequence number indicates the session sequence number of the last trusted semantic fragment. It can be determined using a sliding rollback window, i.e., refreshing the batch to be sent before fragment reset, and then... It equals the current sequence number minus the predetermined number of segments; alternatively, it can be explicitly recorded, whereby the device records the session sequence number of the currently confirmed segment as the last reliable checkpoint after each successful local motion estimation and stripe refresh. After the rollback request is issued, the device will clear the uploaded width to zero and reset the local splicing segments.

[0050] Step 206: Based on the retransmission start number returned by the server, retransmit semantic fragments with in-session sequence numbers not less than the retransmission start number from the historical cache, or send recalculated replacement semantic fragments. Also, send keyframe fragments for re-anchoring the incremental canvas to the server for the reset local spliced ​​segments. After completing ledger truncation and replay, the server returns the retransmission start number, which in this embodiment is... There are two methods for resuming transmission on the device side: One is to retransmit all fragments with sequence numbers no less than the retransmission start number in batches from the historical cache, ordered by session sequence number. In this case, the reconstructed server-side canvas pixels are consistent with those before the rollback. The purpose of the rollback is to bring both the end and cloud back to a defined sequence number baseline, clear the stage identification results and identification trigger baseline derived from untrusted intervals, and force the next segment to be re-anchored with a keyframe. The other method, when the device retains sufficient original data for recalculation, re-executes local splicing and export on fragments after the checkpoint to obtain corrected replacement semantic fragments, which are then sent in place of the original fragments. In this case, the rollback also corrects the canvas pixel content. Both methods share the same ledger truncation and reconstruction process on the server side, and can be implemented depending on whether the device retains the original data. If fragments after a checkpoint in the historical cache have been evicted and cannot be fully replayed, the device abandons the replay and directly re-anchors with new keyframe fragments. The server continues to append to the end of the current valid area, sacrificing a small amount of content in that section to maintain end-cloud consistency, rather than leaving the two ends in a long-term forked state. Regardless of the retransmission method used, the device generates keyframe fragments for the reset new local splicing segment and sends them to the server. The server uses these as re-anchor frames and appends them to the end boundary of the valid area of ​​the incremental canvas, making the new local segment and the previous content form a continuous canvas on the server. Afterward, the device continues to export and send the image strips of the new segment according to steps 202 to 204.

[0051] Steps 201 to 206 above constitute a complete device-side processing scheme. Its working principle is as follows: the device converts the high-frame-rate local image stream into semantic fragments containing only truly new content and with spatial alignment. It provides the server with a replayable event source using session sequence numbers and historical cache. In case of local splicing failure or segment rotation, it negotiates rollback with the server using checkpoint sequence numbers, then resumes transmission with precise retransmission start sequence numbers and re-anchors using keyframes, thereby achieving a balance between network transmission volume, stage result latency, and error recovery costs. The following describes optional improvements to this embodiment.

[0052] In some alternative implementations, since rollback requires the server to perform ledger truncation, canvas reconstruction, and piecewise replay, its cost is significantly higher than a local retry. Therefore, the device can adopt a tiered approach to local stitching failures. Specifically, when the horizontal displacement estimated by local stitching... The absolute value is less than the displacement dead zone At that time, the equipment sets the horizontal displacement to zero and excludes it from the cumulative displacement, while incrementing the zero displacement count. The horizontal displacement estimated between adjacent frames is compared with the incremental width reported as a placement parameter in Example 1. The latter is different; it refers to the number of new columns generated after the cumulative horizontal displacement across multiple frames reaches a threshold. The displacement dead zone is represented by pixels. In this embodiment, it is shown as... Taking 0.35 pixels as an example, this processing is used to suppress the accumulation of pseudo-displacement when the scanning pen is stationary or slightly jittery, avoiding the generation of stripes when there is no actual new content. A stitching failure is only determined when the zero displacement count reaches a predetermined number; in this embodiment, the predetermined number is 5. For the determined stitching failures, the device further distinguishes the causes: stitching failures determined to be caused by low texture or momentary lack of motion, such as insufficient dark pixels within a frame or a velocity prior close to zero, are considered non-fatal failures, and the device only updates the anchor frame used for local stitching without sending a rollback request; for other stitching failures, the device first marks them for retry, re-executes local stitching on the next local image, and only sends a rollback request if the re-execution still fails. This hierarchical processing ensures that rollback only occurs when absolutely necessary, avoiding repeated reconstruction of the end-cloud state due to momentary low texture during normal scanning. It should be noted that the above-mentioned hierarchical processing is not a necessary condition for implementing this method. Without hierarchical processing, the device can also send a rollback request directly each time the splicing fails. In this case, steps 205 and 206 can also restore the consistency between the end and the cloud.

[0053] Optionally, to control the single-frame computing power on low-configuration embedded devices, the motion estimation in step 201 can be divided into multiple computing levels and automatically switched according to the current state: the default level only uses the coarse estimation results of row and column projection; when the consecutive failure count, the number of successful stitchings, or the predicted longitudinal velocity meet the corresponding conditions, it enters the adaptive level containing multiple candidate fine search; when the conditions further deteriorate or the coarse longitudinal displacement exceeds the fine search radius, normalized cross-correlation candidates are then enabled. The evaluation area of ​​the fine search can be limited to the longitudinal middle band of the frame and skips a predetermined width on the left in the horizontal direction. In this embodiment, taking 20% ​​to 86% of the frame height as the evaluation band and skipping 28 pixels on the left is used as an example to eliminate interference from uneven lighting, lens edge distortion, and pixels that have not yet entered the stable imaging area; dark pixels and background pixels can use a fixed ratio of integer weights. In this embodiment, a weight ratio of 4:1 and a dark pixel judgment threshold of 170 are used as an example, and samples are taken at fixed step intervals along the row direction and then accumulated in parallel using single instruction multiple data instructions. The above parameters are all exemplary design parameters and can be adjusted according to the sensor, lens, and lighting conditions.

[0054] In a variation of this embodiment, strip generation, transmission, and recognition can constitute a three-level independently configurable throttling: the first level limits the frequency of strip generation and transmission by a displacement accumulation threshold and a minimum strip upload interval; the second level limits transmission by a batch number threshold or a refresh time threshold; and the third level consists of a server-side time threshold and a content growth threshold. These three levels of throttling apply to acquisition, transmission, and recognition respectively, and adjusting any parameter at any level does not change the correctness of the other two levels.

[0055] Example 4 This embodiment provides an edge-cloud collaborative incremental optical character recognition system for implementing the methods described in Embodiments 1 to 3 above. Figure 1 As shown, the system includes at least a scanning pen device 100, a communication network 200, an incremental adaptation service 300, and an upstream optical character recognition engine 400. The scanning pen device 100 executes the method described in Embodiment 3, converting high-frame-rate local images into semantic segments with placement semantics, and sending them to the incremental adaptation service 300 running on the server via the communication network 200; the incremental adaptation service 300 executes the method described in Embodiment 1 or Embodiment 2, and calls the upstream optical character recognition engine 400 when recognition is required.

[0056] The incremental adaptation service 300 includes a session and sequence verification module 310, a semantic fragmentation ledger 320, a fast canvas module 330, a staged optical character recognition scheduling module 340, a rollback module 350, and a final state input selection module 360. The session and sequence verification module 310 uses a global session table to manage multiple scanning sessions, setting reentrant locks and condition variables for each session to ensure the unique order of application events using session sequence numbers. In the second embodiment, it also verifies canvas generation and predecessor state fingerprints. The semantic fragmentation ledger 320 saves replayable image events according to session sequence numbers. The fast canvas module 330 applies events, maintains the incremental canvas, and updates the valid area. Here, the fast canvas and the incremental canvas refer to the same object. The stage optical character recognition scheduling module 340 determines the recognition trigger condition based on the recognition trigger baseline and calls the upstream optical character recognition engine 400. The rollback module 350 truncates the semantic fragmentation ledger 320 after the checkpoint sequence number, clears the derived state, and replays the retained events. The final state input selection module 360 ​​selects the final state recognition input between the complete mosaic image and the incremental canvas. The above modules can be deployed on the same server or distributed. The incremental adaptation service 300 and the upstream optical character recognition engine 400 can be deployed on the same machine or called through a remote interface.

[0057] The system's end-to-cloud interaction under three scenarios—normal upload, rollback, and result invalidation—is as follows: During the normal upload phase, the scanning pen device 100 sequentially sends keyframe fragments and stripe fragments. The incremental adaptation service 300 accepts these fragments one by one and appends them to the incremental canvas. Stage optical character recognition can be triggered by any successfully accepted semantic fragment. When the scanning pen device 100 sends a rollback request, the original stage recognition result is no longer considered valid. The incremental adaptation service 300 truncates the ledger, reconstructs the canvas, and returns the retransmission start number. The scanning pen device 100 then resumes the upload and re-anchors using keyframes. After scanning, final-state optical character recognition does not rely on the stitching of stage recognition results but is directly performed based on the image selected by the final-state input selection module 360. The result interface allows for long polling with a limited duration. During the waiting period, if there are stage results, final-state results, errors, or cancellation statuses, the latest session state is returned.

[0058] Optionally, each session can record decoding time, receiving and processing time, cumulative time for stage recognition, final state recognition time, number of upstream calls, number of uploaded fragments, and number of uploaded bytes. These metrics can be used to dynamically adjust the compression algorithm, batch size, time threshold, and content growth threshold. In a variation of this embodiment, the incremental adaptation service 300 can also persist session events as session-isolated event log files in an append-only manner, recording fragment reception, stage recognition, rollback, and session end events and their parameters one by one, and saving the input images used for each stage recognition and final state recognition. This persistent record corresponds to the semantic fragment ledger 320 in memory, enabling the canvas state and recognition results at any time to be reproduced and audited offline, and can also be used to replay the location of the cloud state fork at the positioning end. In addition, to accelerate the recovery of long sessions, the semantic sharded ledger 320 can periodically save canvas snapshots with canvas generations, checkpoint numbers, prefix state fingerprints and valid boundaries. During rollback, it can replay subsequent semantic shards starting from the most recent trusted snapshot no later than the checkpoint, instead of having to completely replay from the first retained shard.

[0059] Example 5 This embodiment provides an electronic device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the method described in any one of Embodiments 1 to 3. When the electronic device acts as a server, it can be a public cloud server, an enterprise intranet server, an edge gateway, or a mobile phone. The processor can be a general-purpose central processing unit, or it can work with a graphics processor or a dedicated recognition accelerator to perform optical character recognition. The memory stores a semantic fragmented ledger, an incremental canvas, a recognition trigger baseline, and program instructions for implementing the method described in Embodiment 1 or Embodiment 2. When the electronic device acts as a scanning pen, it can be a scanning pen, a word pen, or a translation pen. The processor is a low-power embedded processor, and the memory stores a local splicing canvas, a history cache, and program instructions for implementing the method described in Embodiment 3. The electronic device may also include a two-dimensional image sensor for acquiring local images, a supplementary lighting unit, a scanning key, and a wireless communication module for communicating with the server. This disclosure also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the method described in any of the above embodiments. The computer-readable storage medium can be a read-only memory, a random access memory, flash memory, a hard disk, or an optical disk, etc.

[0060] It should be noted that the terms "first," "second," etc., used in this specification are only used to distinguish similar objects, and the elements defined by them do not necessarily require or imply any actual relationship or order between these elements, nor are they used to describe a specific order or sequence.

[0061] The terms “comprising,” “including,” or any other variations thereof, as used herein, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0062] The phrase "execute according to a certain element" as used in this specification means at least according to that element, including both "execute only according to that element" and "execute according to that element and other elements".

[0063] Those skilled in the art will understand that variations of the method steps described above can be obtained based on the embodiments of this disclosure without any inventive effort. For example, the sequence numbers of the steps described in the method embodiments of this specification do not themselves constitute a limitation on the execution order of these steps. Unless there is an explicit specific limitation in the context, the steps may be executed in a different order than that shown in the embodiments (e.g., executing the steps with larger sequence numbers first, then the steps with smaller sequence numbers), or they may be executed in parallel. Furthermore, other steps may be inserted between multiple steps with consecutively numbered sequence numbers.

[0064] Those skilled in the art will also understand that the division of modules and units described in the embodiments of this disclosure is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. The modules or units may or may not be physically separate. The components shown as modules or units may or may not be physical modules. They may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the embodiments of this disclosure according to actual needs. In addition, the connection or interaction relationships shown in the accompanying drawings are only illustrative and do not constitute a limitation of this disclosure.

[0065] The specific embodiments described in this specification are merely illustrative examples of this disclosure. For instance, specific numerical values, parameters, protocols, network architectures, component models, drawing scales, and specific shapes shown in the drawings are only examples and do not constitute a limitation of this disclosure. Features of the various embodiments of this disclosure can be arbitrarily combined without contradicting each other, and the solutions formed by such combinations also fall within the protection scope of this disclosure.

[0066] All references to this disclosure are considered to be incorporated herein in their entirety so as to serve as a basis for modification where necessary. Furthermore, it should be understood that, upon reading this disclosure, those skilled in the art may make various alterations or modifications to it, and these equivalent forms also fall within the scope of protection claimed herein.

Claims

1. An edge-cloud collaborative incremental optical character recognition method, applied to the server side, characterized in that, include: The device receives semantic segments from the scanning pen device. The semantic segments carry in-session sequence numbers and include keyframe segments and strip segments. The strip segments also carry image strips and tail append markers. The image strips are newly added column images exported from the local splicing canvas by the scanning pen device and located between the uploaded width and the current effective width. The tail append markers are used to indicate that the image strips have been spatially aligned by the scanning pen device and should be appended to the tail boundary of the effective area of ​​the incremental canvas. When the in-session sequence number equals the expected sequence number, the semantic fragment is accepted, the accepted semantic fragment is written into the replayable semantic fragment ledger according to the in-session sequence number, and the expected sequence number is advanced. The incremental canvas and its effective area are initialized based on the keyframe segments; Based on the tail append mark, the image strip is appended to the tail boundary of the effective region of the incremental canvas, and the effective region is updated according to the width of the image strip; When the recognition trigger condition is met, phased optical character recognition is performed on the image located in the effective area of ​​the incremental canvas, and the recognition trigger baseline is updated. The recognition trigger baseline is the reference quantity on which the recognition trigger condition is determined. In response to a rollback request carrying a checkpoint sequence number from the scanning pen device, semantic fragments with in-session sequence numbers greater than the checkpoint sequence number in the semantic fragment ledger are deleted. The incremental canvas, the recognition trigger baseline, and the stage optical character recognition results derived from the deleted semantic fragments are cleared. The semantic fragments retained in the semantic fragment ledger are replayed in in-session sequence number order to reconstruct the incremental canvas. A retransmission start sequence number determined based on the checkpoint sequence number is returned to the scanning pen device so that the scanning pen device can retransmit the semantic fragments from the retransmission start sequence number.

2. The method according to claim 1, characterized in that, The semantic segments also carry canvas generations; The step of accepting the semantic fragment when the in-session sequence number is equal to the expected sequence number includes: accepting the semantic fragment when the canvas generation carried by the semantic fragment is equal to the current canvas generation and the in-session sequence number is equal to the expected sequence number, and rejecting semantic fragments carrying canvas generations different from the current canvas generation. The method further includes: after completing the replay, establishing a new canvas generation different from the previous canvas generation as the current canvas generation, and returning the new canvas generation to the scanning pen device along with the replay start number.

3. The method according to claim 2, characterized in that, The semantic segment also carries the preceding state fingerprint; The method further includes: for each semantic segment received, calculating a new prefix state fingerprint based on the previous prefix state fingerprint, the current canvas generation, the session sequence number of the semantic segment, the segment type, the placement semantics, and the image payload digest, wherein the placement semantics include the placement parameters of the tail appending marker or the image stripe; The semantic segment is applied to the incremental canvas only if the predecessor state fingerprint carried by the semantic segment is equal to the current prefix state fingerprint. After the replay is completed, the checkpoint prefix state fingerprint obtained from the replay is returned to the scanning pen device as the reference fingerprint for the new canvas generation.

4. The method according to claim 3, characterized in that, The recognition result of the optical character recognition in the aforementioned stage carries a result version, which includes the canvas generation at the time of the recognition, the applied in-session sequence number, and the prefix state fingerprint. The method further includes: outputting the recognition result only when the canvas generation in the result version is equal to the current canvas generation and the prefix state fingerprint in the result version belongs to the current prefix state fingerprint sequence of the semantic shard ledger, and discarding recognition results that do not meet the aforementioned conditions.

5. The method according to claim 1, characterized in that, The recognition trigger baseline includes the execution time of the previous stage of optical character recognition and the content of the effective area when performing this recognition; The identification triggering condition is: the difference between the current time and the execution time reaches a time threshold, and the difference between the current content of the effective area and the content of the effective area when the identification is performed reaches a content growth threshold.

6. The method according to claim 1, characterized in that, Also includes: In response to the end of scanning, if a complete stitched image uploaded by the scanning pen device has been received, the complete stitched image is used as the input for final-state optical character recognition; if the complete stitched image has not been received, the image located within the effective area of ​​the incremental canvas is used as the input for final-state optical character recognition.

7. The method according to claim 1, characterized in that, The semantic segmentation also includes a strip segment to be located, which carries an image strip and placement parameters but does not carry the tail append mark. The placement parameters include the added width and vertical displacement. The method further includes: appending a keyframe fragment received after the effective region of the incremental canvas has been initialized as a re-anchoring frame to the tail boundary of the effective region; and performing a validity reduction on the placement parameters carried by the strip fragment to be positioned before applying it to the incremental canvas, the validity reduction including taking a non-negative value for the added width and clamping the longitudinal displacement within a predetermined range.

8. An edge-cloud collaborative incremental optical character recognition method, applied to a scanning pen device, characterized in that, include: Local stitching is performed on partially overlapping images acquired in successive acquisitions to form a local stitching canvas; Starting from the uploaded width, and taking the difference between the current effective width of the local splicing canvas and the uploaded width as the strip width, export the newly added column image from the local splicing canvas as the image strip; Semantic segments are generated and saved in a history cache indexed by session sequence number. The semantic segments carry session sequence number. The semantic segments include keyframe segments generated at the beginning of local splicing segmentation, carrying the starting segment image of the local splicing canvas, and strip segments carrying the image strips and tail append markers. The tail append markers are used to indicate that the image strips have completed local spatial alignment and should be appended to the tail boundary of the effective area of ​​the incremental canvas maintained by the server. The semantic fragment is sent to the server, and after successful transmission, the uploaded width is updated to the current valid width. When local splicing fails and the preset rollback conditions are met, or when the local splicing canvas reaches its capacity limit, a rollback request carrying a checkpoint sequence number is sent to the server, and the uploaded width is cleared to zero and the local splicing segment is reset. The checkpoint sequence number indicates the session sequence number of the last trusted semantic segment. Based on the retransmission start sequence number returned by the server, retransmit semantic fragments with in-session sequence numbers not less than the retransmission start sequence number from the historical cache, or send recalculated replacement semantic fragments, and send keyframe fragments for re-anchoring the incremental canvas to the server for the reset local splicing segment.

9. The method according to claim 8, characterized in that, The method further includes determining whether the local splicing failure meets the preset rollback condition in the following manner: When the absolute value of the horizontal displacement estimated by local splicing is less than the displacement dead zone, the horizontal displacement is set to zero and the zero displacement count is accumulated. When the zero displacement count reaches a predetermined number, it is determined that a splicing failure has occurred. For stitching failures determined to be caused by low texture or momentary lack of motion, only the anchor frames used for local stitching are updated without sending the rollback request; For any remaining stitching failures, first re-execute the local stitching once for the next local image. If the re-execution still fails, send the rollback request.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 9.

Images