A method for scanning a code and an electronic device

Abstract

The application discloses a code scanning method and an electronic device. The application relates to the technical field of terminals. In the method, the electronic device can determine whether to issue an adjustment instruction according to an original image. That is to say, the electronic device does not need to determine whether the camera parameter needs to be adjusted through an application side, but can determine whether the camera parameter needs to be adjusted through a hardware abstraction layer, and then sends an adjustment instruction to a hardware layer through the hardware abstraction layer, so that the time length for identifying a bar code by the application side is greatly shortened.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a method and electronic device for scanning codes. Background Technology

[0002] In barcode scanning scenarios, the distance between the electronic device and the barcode may be too great, or the barcode itself may be too small, resulting in the electronic device being unable to recognize the barcode in the image it captures. In such cases, the electronic device may need to adjust its camera parameters to enable barcode recognition. How to achieve faster barcode recognition is a pressing problem that needs to be solved in this field. Summary of the Invention

[0003] This application provides a barcode scanning method and an electronic device. The electronic device no longer needs to determine whether camera parameters need to be adjusted through the application side. Instead, it can determine whether camera parameters need to be adjusted through the HAL layer and then send adjustment instructions to the hardware layer through the HAL layer, which greatly shortens the time for the application side to recognize barcodes.

[0004] In a first aspect, this application provides a barcode scanning method applied to an electronic device, the electronic device comprising: a barcode scanning application, a Hardware Abstraction Layer (HAL) layer, and a hardware layer. The method comprises: in response to activating the barcode scanning function of the barcode scanning application, acquiring a first original image using first camera parameters through the hardware layer, the first original image containing a barcode; generating a first target image based on the first original image through the HAL layer, and concurrently performing an action of determining first target camera parameters based on the first original image through the HAL layer; transmitting the first target camera parameters to the hardware layer through the HAL layer; acquiring a second original image using the first target camera parameters through the hardware layer, the second original image containing a barcode; generating a second target image based on the second original image through the HAL layer; and obtaining information carried by the barcode in the second target image through the barcode scanning application.

[0005] By implementing the first approach, the HAL layer can directly determine whether camera parameters need to be adjusted based on the original image, without waiting for application B to fail to recognize the barcode in the target image before determining whether camera parameters need to be adjusted. In this way, the hardware layer only needs to wait a short period of time to complete the adjustment of camera parameters, and then the barcode can be recognized successfully.

[0006] Specifically, the HAL layer determines whether camera parameters need adjustment based on the original image of frame t and then issues an adjustment command. At this time (i.e., the moment the adjustment command is issued), the application layer attempts to recognize the barcode in the target image of frame t, but fails. After receiving the adjustment command, the hardware layer adjusts the camera parameters. Since adjusting camera parameters also takes some time, after the adjustment is complete, the hardware layer can send the original image with adjusted camera parameters to the HAL layer, which then sends the target image with adjusted camera parameters to application B. At this time (i.e., the moment application B receives the target image with adjusted camera parameters), it is already frame t+n. Here, n can be the ratio of the hardware layer's adjustment time to the duration of a single frame. In other words, when application B fails to recognize the barcode in the target image of frame t, it only needs to wait until it obtains the image of frame t+n to successfully recognize the barcode. The overall cycle is short, resulting in a better user scanning experience.

[0007] In conjunction with the first aspect, in some embodiments, after generating a first target image based on the first original image through the HAL layer and performing the action of determining the first target camera parameters based on the first original image through the HAL layer in parallel, the method further includes: transmitting the first target image to the barcode scanning application through the HAL layer; identifying the barcode in the first target image through the barcode scanning application; and adjusting the camera parameters of the electronic device to the first target camera parameters through the hardware layer when identifying the barcode in the first target image through the barcode scanning application. For example, refer to... Figure 5 The two steps S216a and S215 in the process.

[0008] In this way, generating the target image M1 based on the HAL layer and determining whether camera parameters need to be adjusted are executed in parallel, and S215 and S216a can also be executed in parallel. That is to say, the hardware layer does not need to wait for application B to determine whether camera parameters need to be adjusted before adjusting the camera parameters.

[0009] In conjunction with the first aspect, in some embodiments, transmitting the first target image to the scanning application through the HAL layer specifically includes: in response to generating the first target image based on the first original image through the HAL layer, sending the first target image to the scanning application through the HAL layer; transmitting the first target camera parameters to the hardware layer through the HAL layer specifically includes: in response to analyzing the first original image through the HAL layer, transmitting the first target camera parameters to the hardware layer through the HAL layer.

[0010] In conjunction with the first aspect, in some embodiments, the method further includes: obtaining second feature information of the barcode in the second original image through the HAL layer, wherein the action of obtaining the second feature information of the barcode in the second original image through the HAL layer and the action of generating a second target image based on the second original image through the HAL layer are performed in parallel; the information carried by the barcode in the second target image obtained through the barcode scanning application specifically includes: obtaining the information carried by the barcode in the second target image based on the second feature information through the barcode scanning application.

[0011] The aforementioned second feature information includes the location, category, and feature points of the barcode in the second target image. The HAL layer can acquire the barcode's feature information, such as its location, category, and feature points. Based on this feature information, the HAL layer can determine whether camera parameters need to be adjusted. Furthermore, it can send the aforementioned information to the corresponding application simultaneously with the original image, thereby reducing application processing time and improving application processing efficiency.

[0012] In conjunction with the first aspect, in some embodiments, the second feature information includes a first feature point of the barcode in the second original image, the first feature point being in a distorted state, and the information carried by the barcode in the second target image obtained by the scanning application based on the second feature information specifically includes: performing a correction process on the first feature point to obtain the corrected first feature point; and obtaining the information carried by the barcode in the second target image based on the corrected first feature point.

[0013] In this way, barcode recognition can be completed even in scenarios where the barcode is scanned at a small angle.

[0014] In conjunction with the first aspect, in some embodiments, determining the parameters of the first target camera based on the first original image through the HAL layer specifically includes: obtaining first feature information of the barcode in the first original image through the HAL layer; and determining the parameters of the first target camera based on the first feature information of the barcode in the first original image through the HAL layer. The first feature information includes information such as the position, category, and feature points of the barcode in the first target image.

[0015] In conjunction with the first aspect, in some embodiments, before obtaining the first feature information of the barcode in the first original image through the HAL layer, the method further includes: cropping the first original image through the HAL layer to obtain a first cropped image, the first cropped image including the barcode; obtaining the first feature information of the barcode in the first original image through the HAL layer specifically includes: obtaining the first feature information of the barcode in the first original image based on the first original image and the first cropped image through the HAL layer. For example, the cropped image can be obtained through a pre-screening module.

[0016] This increases the proportion of the barcode in the image, thereby improving the success rate of the subsequent barcode information detection module in obtaining the barcode's feature information.

[0017] In conjunction with the first aspect, in some embodiments, cropping the first original image through the HAL layer specifically includes: determining a first prior box from a preset prior box based on the feature information of the barcode in the third original image through the HAL layer, the third original image being acquired before the first original image; cropping the area where the first prior box is located in the first original image through the HAL layer to obtain the first cropped image.

[0018] The aforementioned pre-screening module is deployed in the HAL layer. The HAL layer can crop the area where the barcode may exist in the current frame's original image based on the barcode position detection results in the previous frame's original image, resulting in a cropped original image. This increases the proportion of the barcode in the image, improves the success rate of the barcode information detection module in obtaining the barcode's feature information, and increases the maximum recognition distance for scanning.

[0019] In conjunction with the first aspect, in some embodiments, the first target camera parameter includes a first target zoom ratio. Determining the first target camera parameter based on the first original image using the HAL layer specifically includes: determining a first proportion of the barcode in the second original image using the HAL layer; determining the first target zoom ratio using the HAL layer based on the first proportion; and, when the position of the electronic device remains unchanged, having a barcode proportion greater than the first proportion in the original image acquired using the first target zoom ratio. For example, refer to the following description of the zoom determination module.

[0020] In conjunction with the first aspect, in some embodiments, the first target camera parameters include first target exposure parameters, and determining the first target camera parameters based on the first original image through the HAL layer specifically includes: determining the first grayscale value of all or part of the area where the barcode is located in the second original image through the HAL layer; and determining the first target exposure parameters through the HAL layer based on the first grayscale value.

[0021] In conjunction with the first aspect, in some embodiments, the first target camera parameters include first target focus parameters. Determining the first target camera parameters based on the first original image through the HAL layer specifically includes: determining the first coordinates of the barcode center in the second original image through the HAL layer; and determining the first target focus parameters based on the first coordinates through the HAL layer. For example, refer to the following description of the 3A judgment module and... Figure 11D Related information.

[0022] In conjunction with the first aspect, in some embodiments, the method further includes: acquiring the third raw image using the first camera parameters through the hardware layer; generating a third target image based on the third raw image through the HAL layer; and discarding the third target image through the barcode scanning application; wherein the acquisition of the third raw image occurs after the acquisition of the first raw image and before the acquisition of the second raw image. For example, the third raw image may refer to... Figure 5 The original image m2 shown, and the third target image can refer to... Figure 5 The target image M2 is shown.

[0023] In conjunction with the first aspect, in some embodiments, the method further includes: discarding the first target image through the scanning application. If the first target image is... Figure 5 If the target image M2 shown is discarded by the electronic device via a barcode scanning application, then the electronic device will discard the first target image.

[0024] In a second aspect, this application provides an electronic device including a memory, a processor, and computer instructions stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect or any embodiment of the first aspect.

[0025] Thirdly, this application provides a chip system including a processing circuit and an interface circuit. The interface circuit is used to receive code instructions and transmit them to the processing circuit. The processing circuit is used to allow the code instructions to execute the method described in the first aspect or any embodiment of the first aspect.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a processor, implements the method described in the first aspect or any of the embodiments of the first aspect.

[0027] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when run on a processor, implements the method described in the first aspect or any of the embodiments of the first aspect.

[0028] The benefits of the second to fifth aspects can be referenced from the first aspect mentioned above. Attached Figure Description

[0029] Figure 1 This application provides an example of an application scenario.

[0030] Figure 2 A timing diagram of the interactive process for scanning a QR code is provided for an embodiment of this application;

[0031] Figure 3A This is a flowchart illustrating the process of application A failing to parse a barcode, as provided in an embodiment of this application.

[0032] Figure 3B A schematic diagram illustrating the process of application A failing to detect barcode position in an embodiment of this application;

[0033] Figure 4 A software framework diagram of the electronic device 100 provided in the embodiments of this application;

[0034] Figure 5 A timing diagram of the interactive process for scanning a QR code is provided for an embodiment of this application;

[0035] Figures 6A-6C Schematic diagrams illustrating several parallel execution methods provided in embodiments of this application;

[0036] Figure 7A Execution provided for embodiments of this application Figure 2 The timing diagram of the scanning method shown is shown below;

[0037] Figure 7B Execution provided for embodiments of this application Figure 5 The timing diagram of the scanning method shown is shown below;

[0038] Figure 8 A schematic diagram of a neural network model based on the CSP-PAN structure provided in this application embodiment;

[0039] Figure 9A This is a schematic diagram of the morphological structure of the QR Code provided in the embodiments of this application;

[0040] Figure 9B This is a schematic diagram illustrating the correction of feature points of the acquired barcode, provided as an embodiment of this application.

[0041] Figure 10 This is a schematic diagram of the process for obtaining feature points of a barcode, provided in an embodiment of this application.

[0042] Figure 11A A schematic diagram illustrating the setting of a priori bounding box, provided for an embodiment of this application;

[0043] Figure 11B This is a schematic diagram showing the possible locations of barcodes in the original image of frame t-1.

[0044] Figure 11C A schematic diagram illustrating the input and output information of the barcode information detection module provided in this embodiment of the application;

[0045] Figure 11D This is a schematic diagram of the area where the barcode is located in the original image provided in the embodiments of this application;

[0046] Figure 12 This is a schematic flowchart of a barcode scanning method provided in an embodiment of this application;

[0047] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions “a,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one or more (including two). “At least one of the following” or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0049] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0050] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0051] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0052] First, let me introduce some of the terms and concepts used in the embodiments of this application.

[0053] Barcodes are used to store information (referred to as barcode information), and they can include one-dimensional barcodes (also known as 1D codes) and two-dimensional barcodes (also known as 2D codes). Barcodes are not limited to these types; they can include many other types as well.

[0054] A QR code is a form of encoding that uses alternating black and white images on a two-dimensional plane to record data information (referred to as QR code information) based on specific geometric patterns and rules.

[0055] Scanning a barcode can be understood as the process by which an electronic device captures an image containing a barcode, parses the barcode in the image, and obtains the barcode information.

[0056] Camera parameters can include one or more of the following: exposure parameters, color temperature, focus, focal length, etc. Exposure parameters can further include: aperture, shutter speed, and ISO. The camera parameters of electronic devices affect the raw image captured by the electronic device. Auto Exposure (AE), Auto White Balance (AWB), and Auto Focus (AF) can be collectively referred to as the 3A. AE adjusts the camera's exposure parameters, AWB adjusts the camera's color temperature, and AF adjusts the camera's focus position. Electronic devices can achieve better shooting results by adjusting the 3A.

[0057] Zooming refers to changing the focal length of a lens, thereby altering the shooting angle and the size of the subject. Zooming can include optical zoom, digital zoom, hybrid zoom, and more. Zooming is distinct from focusing.

[0058] Figure 1 An example application scenario provided by an embodiment of this application is illustrated.

[0059] refer to Figure 1 Users can scan barcodes using electronic devices. For example, the electronic device first captures an image using a camera. Then, the electronic device obtains the barcode, such as a QR code, contained within the image. Finally, the electronic device parses the barcode to obtain the information it contains.

[0060] However, in some application scenarios, such as parking payment and ordering food in a queue, the distance between the electronic device and the barcode is too far, or the barcode itself is too small, resulting in the electronic device being unable to parse the barcode in the image it captures, thus affecting the user experience.

[0061] The following describes a QR code scanning interaction process provided by an embodiment of this application.

[0062] Figure 2 An exemplary timing diagram of a QR code scanning interaction process provided in an embodiment of this application is shown.

[0063] The electronic device includes application A, a hardware abstraction layer (HAL layer), and a hardware layer. Application A is an application with barcode scanning functionality installed in the electronic device. Application A can be a system application, such as a camera application, or a third-party application, such as other barcode scanning applications. In this embodiment, application A is not limited.

[0064] Figure 2 The interaction sequence diagram shown is mainly divided into three parts:

[0065] S101-S109: Trigger acquisition of raw image;

[0066] S110a-S110e: Continuously generate target images;

[0067] S114a-S119: Application A is used to identify barcodes in target images.

[0068] Figure 2 The interaction flow sequence diagram shown specifically includes:

[0069] S101. Application A starts the barcode scanning function.

[0070] S102. Application A displays the QR code scanning interface.

[0071] S103-1. Application A calls the openCamera interface to send a request to the HAL layer to start the camera.

[0072] S103-2. The HAL layer sends a request to the hardware layer to start the camera.

[0073] S104. Start the camera at the hardware level.

[0074] S105. The hardware layer returns information indicating that the camera startup is complete.

[0075] S106. Apply A to start the preview function.

[0076] S107. Application A calls the startPreview interface to send a request to the HAL layer to acquire the original image.

[0077] S108.HAL layer configuration parameters and initialization process.

[0078] The S109.HAL layer sends a request to the hardware layer to acquire the raw image.

[0079] In some implementations, after the hardware layer receives a request to acquire the original image, the hardware layer can periodically and continuously acquire the original image, and the HAL layer can also periodically generate the target image.

[0080] For example, after the hardware layer receives a request to acquire the raw image, step group SG11 (S110a-S113a) can be executed.

[0081] S110a. Hardware layer acquires the original image m1.

[0082] S111a. The hardware layer sends the raw image m1 to the HAL layer.

[0083] The target image M1 is generated by the S112a.HAL layer.

[0084] The S113a.HAL layer sends the target image M1 to application A.

[0085] The electronic device can generate the target image M1 through the above steps SG11.

[0086] When the HAL layer sends the target image M1 to application A, application A can process the target image M1 since it is in an idle state. Therefore, application A receives the target image M1.

[0087] After executing S110a-S113a above, proceed with step group SG12 (S110b-S113b).

[0088] S110b. Hardware layer acquires the original image m2.

[0089] S111b. The hardware layer sends the raw image m2 to the HAL layer.

[0090] The S112b.HAL layer generates the target image M2.

[0091] The S113b.HAL layer sends the target image M2 to application A.

[0092] Similarly, the electronic device can generate the target image M2 through the above-described step group SG12.

[0093] When the HAL layer sends the target image M2 to application A, application A is busy executing S114a and cannot process the target image M2. Therefore, application A discards the target image M2.

[0094] Similarly, after executing S110b-S113b above, execute step group SG13 (S110c-S113c), step group SG14 (S110d-S113d), and step group SG15 (S110e-S113e) again. The specific steps contained in step group SG13, step group SG14, and step SG15 can be referred to step group SG11 above.

[0095] It is worth noting that, Figure 2 Each of the above step groups is used to generate one frame of the target image. That is, the execution time of each step group is the single frame duration set by the electronic device. In other words, the electronic device can generate one frame of the target image after executing each step group. However, whether the application side receives the target images sent by the HAL layer depends on whether the application side is currently idle. If the application side is idle, it receives the target images; if the application side is busy, it discards the target images. For example, application A receives target images M1, M3, and M5, and discards target images M2 and M4.

[0096] The steps SG11 to SG15 described above are merely illustrative and are not limited to specific steps. Figure 2 The five frames shown illustrate how the electronic device can generate multiple target images after starting to acquire the original images. Furthermore, steps SG11-SG15 are executed by the hardware layer and HAL layer and do not affect the execution of application A within the corresponding time period.

[0097] In some implementations, after receiving the target image, application A can process the target image and identify the barcode in the target image to obtain barcode information.

[0098] Application A can identify barcodes in the following ways:

[0099] First, application A is used to obtain the location of the barcode in the target image.

[0100] If application A does not find any barcode in the target image, it determines that the recognition has failed, ends the current recognition process, and waits for the next frame of the target image to be reported.

[0101] If application A successfully locates the barcode in the target image, the barcode is then further parsed. Barcode parsing can specifically include: barcode correction, noise reduction, and obtaining barcode information according to the barcode encoding rules.

[0102] If the barcode is successfully parsed, the barcode information can be obtained, meaning the barcode has been successfully recognized. Optionally, application A can also directly jump to the interface corresponding to the barcode information. Optionally, after successfully parsing the barcode, application A can disable the scanning function.

[0103] If parsing fails, recognition is deemed to have failed. Further analysis can determine if zooming is necessary to increase the likelihood of successful barcode recognition in subsequent steps. For example, if the barcode is too small, zooming is required. Application A then calls the callback function provided by the camera framework, issues a zoom command, ends the current recognition process, and waits for the next frame of the target image to be reported.

[0104] In other words, successful barcode recognition includes parsing the barcode and obtaining its information. Failed barcode recognition includes either not finding the barcode or failing to parse the barcode.

[0105] The following is combined with Figures 3A-3B This section describes two scenarios where barcode recognition fails. For example, application A includes a location detection module and a parsing module.

[0106] Figure 3A An exemplary flowchart illustrates the process when application A fails to parse a barcode.

[0107] refer to Figure 3AApplication A acquires the target image. Then, application A inputs the target image to the position detection module to determine the position of the barcode within the target image. After determining the barcode position, application A crops the area containing the barcode from the target image to obtain a cropped image. Application A then inputs the cropped image to the parsing module for parsing. If parsing fails, application A issues a zoom command. Application A waits for the next frame of the target image and repeats the barcode recognition process. For example, application A acquires a target image M1 containing QR code a1, determines the position of QR code a1 in the target image M1 using the position detection module, and then attempts to parse QR code a1, but the parsing fails.

[0108] In other words, the zoom command is only issued by application A after it has completed a barcode recognition process and determined that the parsing has failed. Only then can the hardware layer make corresponding adjustments based on the zoom command. However, most applications A with barcode scanning capabilities currently use various barcode recognition algorithms that typically take tens to hundreds of milliseconds to complete a single barcode recognition process. Furthermore, during the barcode recognition process of application A, the HAL layer continuously sends target images to application A. However, because application A is busy (e.g., in the process of barcode recognition), many target images sent by the HAL layer to application A are discarded, such as target images M2 and M4 mentioned above.

[0109] Figure 3B An exemplary flowchart illustrates the process of application A failing to detect barcode position.

[0110] refer to Figure 3B Application A acquires the target image. Then, application A inputs the target image to the position detection module to determine the position of the barcode within the target image. Application A fails to acquire the barcode position. When the barcode occupies a small portion of the target image, it is more likely to fail to acquire the barcode position. Application A waits for the next frame of the target image and repeats the barcode recognition process. For example, application A acquires a target image M1 containing QR code a2. Because QR code a2 is too small, its position in target image M1 cannot be determined.

[0111] In other words, a zoom command can only be issued after the barcode's position has been detected. If barcode position detection fails, application A will not issue a zoom command, and consequently, the hardware layer will not make any adjustments. If application A fails to detect the barcode's position because the barcode's proportion in the target image is too small, then since the hardware layer will not perform zoom processing, it will be impossible to improve the barcode's proportion in the target image, potentially resulting in continuous recognition failures.

[0112] The following S114a to S117 can be scenarios where a zoom command is issued after the barcode parsing fails.

[0113] S114a. Apply A to identify the barcode in the target image M1.

[0114] For example, after executing S113a, application A obtains the target image M1 and executes S114a.

[0115] S115a. After recognition fails, application A determines whether to zoom.

[0116] After executing S114a, application A can identify the barcode in target image M1, but parsing the barcode in target image M1 fails. Application A can further determine whether zooming is needed based on target image M1. If zooming is determined to be needed, application A can issue a zoom command. The specific zoom command issued by application A includes S116-1 and S116-2.

[0117] S116-1. Application A calls the callback function to send a zoom command to HAL.

[0118] S116-2. The HAL layer sends zoom commands to the hardware layer.

[0119] S117. Zoom processing is performed at the hardware layer.

[0120] After receiving the zoom command, the hardware layer can execute S117. For example, as shown... Figure 2 As shown, the hardware layer completes S117 before executing S110e. That is, the hardware layer acquires the original image m5 after zooming. The remaining original images m1, m2, m3, and m4 are acquired before zooming.

[0121] The following S114b to S115b can also be used to issue zoom commands after the above barcode parsing fails.

[0122] S114b. Apply A to identify the barcode in the target image M3.

[0123] For example, after executing S113a, application A obtains the target image M3 and executes S114a.

[0124] S115b. After recognition fails, application A determines whether to zoom.

[0125] For example, S114a and S115a can be referred to the above S114a and S115a, and will not be repeated here.

[0126] S118 to S119 below can be the case where the above parsing is successful and the barcode information is obtained.

[0127] S118. Apply A to identify the barcode in the target image M5.

[0128] S119. Application A obtains barcode information.

[0129] After obtaining the barcode information, application A can perform related operations. For example, if the barcode information includes a link to a page, the electronic device can navigate to and display the page indicated by the barcode information.

[0130] from Figure 2 As shown in the timing diagram, if application A fails to recognize the barcode in the target image, it needs to determine whether the parameters need to be adjusted. The hardware layer needs to wait for a relatively long time to complete the adjustment of the camera parameters before it can successfully recognize the barcode.

[0131] Specifically, after application A fails to recognize the barcode in the target image of frame t, it issues a zoom command. Since barcode recognition takes a certain amount of time, by this time (i.e., the moment the zoom command is issued), the HAL layer has already acquired the target image of frame t+m. Here, m can be the smallest integer greater than the ratio of application A's recognition time to the duration of a single frame. For example, the recognition time of application A can be referenced... Figure 2 The diagram shows the waiting time for application-side recognition. After receiving the zoom command, the hardware layer performs zoom processing. Since zoom processing also requires a certain amount of time, after completing the zoom processing, the hardware layer can send the zoomed original image to the HAL layer, and the HAL layer will then send the zoomed target image to application A. At this point (i.e., the moment application A receives the zoomed target image), it is already frame t+m+n. Here, n can be the smallest integer greater than the ratio of the hardware layer adjustment time to the single frame duration. For example, the hardware layer adjustment time can be referenced... Figure 2 The diagram shows the waiting time for hardware layer adjustments. This means that when application A fails to recognize the barcode in the target image of frame t, it needs to wait until the image of frame t+m+n is acquired before it can successfully recognize the barcode. The overall cycle is long, resulting in a poor user scanning experience.

[0132] For example, such as Figure 2 As shown, if an electronic device fails to recognize the barcode in the first frame image, it may need to wait until the fifth frame image is acquired before it can successfully recognize the barcode.

[0133] For example, suppose it takes 100ms for the user to activate the scanning function and trigger the camera, for example, executing steps S101 to S103-2. It takes 500ms for the electronic device to start capturing the raw image, for example, executing steps S104 to S109. The frame rate for image capture is set to 30ms, for example, the execution time of step group SG1 is 30ms. The time for the application to successfully recognize the barcode is 40ms. The time for the application to fail to recognize the barcode is greater than 40ms. Thus, it may take 790ms from the user activating the scanning function to the electronic device successfully acquiring the barcode information. This results in a long waiting time for the user and a poor scanning experience.

[0134] To address the aforementioned issues, this application provides a barcode scanning method. In this method, the electronic device can determine whether an adjustment command needs to be issued based on the original image. In other words, the electronic device no longer needs to determine whether camera parameters (e.g., zoom) need adjustment through the application side; instead, it can determine this through the HAL layer and then send the adjustment command to the hardware layer, significantly reducing the time required for the application side to recognize the barcode.

[0135] Furthermore, the HAL layer's step of determining whether camera parameters need adjustment does not affect the application's acquisition of the target image. In other words, the HAL layer can generate the target image and determine whether camera parameters need adjustment in parallel. This allows the application to display the acquired target image at the correct frame rate and recognize the QR code within it. The application can also shorten the time required to recognize the barcode in the target image, enabling electronic devices to acquire barcode information faster, allowing users to understand the barcode information more quickly, and providing a better user scanning experience.

[0136] Before introducing the scanning method provided in the embodiments of this application, the software structure of the electronic device involved in the embodiments of this application will be introduced first.

[0137] Figure 4 An exemplary software framework diagram of an electronic device 100 provided in an embodiment of this application is shown.

[0138] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture system as an example to exemplify the software structure of electronic device 100.

[0139] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the electronic device system is divided into four layers, from top to bottom: the application layer, the system framework layer, the hardware abstraction layer, and the hardware layer.

[0140] The application layer can include a series of application packages. For example... Figure 4As shown, the application package can include applications such as camera apps, QR code scanning apps, and gallery apps.

[0141] Camera applications can refer to system applications of electronic devices, specifically applications used to manage camera services.

[0142] A barcode scanning app can refer to any application that has barcode scanning functionality. These apps can be system applications or third-party applications, such as social media apps, lifestyle apps, and so on. A barcode scanning app can utilize the camera service of an electronic device through a camera app; that is, the electronic device can use the barcode scanning app to recognize barcodes in real-time captured images.

[0143] Image libraries store images. Beyond recognizing barcodes in real-time acquired target images, electronic devices can also use barcode scanning applications to recognize barcodes in stored images in other scenarios.

[0144] In the embodiments of this application, the names of the various applications mentioned above are for illustrative purposes only and do not limit their functions.

[0145] The application layer can also provide a user interface for interaction with the user. Electronic devices can initiate the scanning function through user actions on the user interface, thereby displaying the scanning interface. For example, the scanning interface may include a preview interface, a flash control, etc. The application layer not only handles the basic decoding process but also integrates functions such as intelligent scene recognition and machine learning. For example, the application layer may include a business logic module for barcode recognition.

[0146] In some implementations, the barcode scanning application is used to obtain the target image sent by the HAL layer, to identify the barcode in the target image, and to obtain barcode information from the target image.

[0147] The system framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes predefined functions. For example, it includes a camera framework, an image processing framework, and various APIs. The framework layer provides functionality and interfaces for upper-layer applications.

[0148] The Hardware Abstraction Layer (HAL) acts as a bridge between the hardware of an electronic device and its operating system. The HAL can encapsulate drivers for sensors from different manufacturers, providing a consistent interface for upper layers. This allows upper-layer software to be free from the implementation details of the underlying hardware, improving code portability and reusability. In this embodiment, the specific implementation of the HAL layer in camera functionality is used as an example. The HAL layer can specifically include a camera HAL layer (also known as a webcam HAL layer).

[0149] The HAL layer is used to generate a target image based on the original image. Specifically, after receiving the original image, the HAL layer converts it into the target image required by the application layer. For example, the HAL can receive the resolution and pixel format indicated by the application layer and generate a target image with the indicated resolution and pixel format based on the original image.

[0150] In the embodiments of this application, the HAL layer is not limited to adjusting the resolution and pixel format. It can also perform more specific processing on the original image, such as noise reduction and contrast enhancement, to obtain the target image. There are no limitations on this.

[0151] The HAL layer is also used to send the target image to the application layer. For example, the HAL layer can define a standardized interface (such as ICameraDeviceSession) through HIDL or AIDL to encapsulate the target image into a GraphicBuffer object and pass it to the application layer through ANativeWindowBuffer.

[0152] In this embodiment, the HAL layer includes: a pre-screening module, a barcode information detection module, and a camera behavior judgment module.

[0153] The barcode information detection module includes modules such as a barcode location detection module and a barcode feature point detection module. The barcode information detection module is used to determine the feature information of the barcode in the original image. The barcode feature information includes one or more of the following: barcode location, category, feature points, etc.

[0154] The pre-screening module is used to predict the areas where the barcode may be present in the original image.

[0155] The camera behavior determination module is used to determine whether camera parameters need adjustment and whether corresponding adjustment commands need to be sent to the hardware layer. This module includes modules such as a zoom determination module and a 3A determination module. The zoom determination module determines whether zooming is needed and, if so, the target zoom ratio. The 3A determination module determines whether camera parameters such as exposure, color temperature, and focus need adjustment. Exposure, color temperature, and focus are collectively referred to as 3A parameters. Once the camera behavior determination module determines that hardware layer adjustments are required, it sends corresponding commands (such as zoom commands, 3A adjustment commands, etc.) to the camera driver, causing the hardware layer to make the necessary adjustments and thus improve image quality.

[0156] It's worth noting that since most barcodes are black and white, the size of the barcode in the original image has a significant impact on barcode recognition. In other words, the focal length is the camera parameter that has the greatest impact on barcode recognition. Based on this, the camera behavior judgment module primarily determines whether zooming is needed and the target zoom ratio. Optionally, other camera parameters may also have some impact on barcode recognition, and the camera behavior judgment module can also determine whether other camera parameters need to be adjusted; this is not limited.

[0157] For more detailed information on the pre-screening module, barcode information detection module, and camera behavior judgment module mentioned above, please refer to the subsequent introductions, which will not be elaborated here.

[0158] Optionally, a kernel layer may be included between the HAL layer and the hardware layer of the electronic device. The kernel layer includes camera drivers, display drivers, audio drivers, and sensor drivers, etc.

[0159] The hardware layer includes various hardware devices within an electronic device. The hardware responsible for the camera system within this layer can be referred to as the camera itself. Camera hardware can include a camera module, image processor, GPU, NPU, and so on. The camera module can include one or more lens modules, one or more image sensors, an image signal processor (ISP), and so on. Optionally, the camera module may also include a time-of-flight (TOF) sensor, a multispectral sensor, and so on. The lens module is the core component of the camera module, responsible for focusing light and imaging. A lens module can include one or more of the following lenses: main camera, telephoto lens, macro lens, ultra-wide-angle lens, and depth-of-field lens. The telephoto lens is used to capture distant objects and enables optical zoom.

[0160] The hardware layer is used to acquire raw images. Specifically, an object's optical image is projected onto a photosensitive element through a lens. This photosensitive element can be a CCD or a CMOS phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the ISP (Image Signal Processor) for conversion into a digital image signal, i.e., the raw image. For example, the raw image can be unprocessed Bayer GRBG format image data. The hardware layer also sends the raw image to the HAL (Hardware Alignment Layer) layer via a hardware interface. The hardware layer can also adjust the lens and focal length according to adjustment commands issued by the HAL layer.

[0161] It is understandable that both the original image and the target image can contain image data acquired by electronic devices, but their functions, data formats, and processing methods differ.

[0162] In the embodiments of this application, Figure 4The software framework shown is for illustrative purposes only. The software framework of an electronic device may also include other forms of software framework, and there is no limitation thereto.

[0163] In combination with the above Figure 4 The following describes the internal interaction process for scanning a QR code, as provided in an embodiment of this application.

[0164] Figure 5 An exemplary timing diagram of the internal interaction process of scanning a code is shown in an embodiment of this application.

[0165] The electronic device includes application B, a hardware abstraction layer (HAL layer), and a hardware layer. Application B is an application with barcode scanning functionality installed in the electronic device. Application B can be a system application, such as a camera application, or a third-party application, such as other barcode scanning applications. In this embodiment, application B is not limited. For detailed descriptions of application B, the HAL layer, and the hardware layer, please refer to the above. Figure 4 The relevant information will not be elaborated here.

[0166] Figure 5 The interaction sequence diagram shown is mainly divided into three parts:

[0167] S201-S209: Trigger acquisition of raw images;

[0168] S210a-S213c: Continuously generate target images and continuously determine whether camera parameters need to be adjusted;

[0169] S216a-S216b: Application B is used to identify barcodes in target images.

[0170] Figure 5 The interaction flow sequence diagram shown specifically includes:

[0171] S201. Application B starts the barcode scanning function.

[0172] Typically, for privacy and security reasons, to prevent application B from unauthorized access to and use of the electronic device's camera hardware, the electronic device can initiate a QR code scanning function through application B after receiving a user action. The user action can include one or more of the following: touch operation, click operation, gesture operation, knuckle operation, voice operation, and eye-tracking operation. For example, the user action could refer to clicking the QR code scanning control provided in application B.

[0173] In some implementations, before executing S201, the electronic device may launch application B in response to a user operation, and then execute S201. The electronic device may also launch application B and the QR code scanning function simultaneously. For example, the electronic device may respond to a control on the desktop used to launch the QR code scanning function of application B and execute S201. In this embodiment, the specific method of executing S201 is not limited.

[0174] S202. Application B displays the QR code scanning interface.

[0175] In some implementations, after application B initiates the QR code scanning function, the electronic device can launch the camera application. Application B can then use the camera services provided by the electronic device through the camera application.

[0176] After executing S201 above, the electronic device can display the QR code scanning interface provided by application B. At this time, the electronic device can trigger the camera to start. Specifically, the electronic device triggering the camera to start can include S203-1 and S203-2.

[0177] S203-1. Application B calls the openCamera interface to send a request to the HAL layer to start the camera.

[0178] S203-2. The HAL layer sends a request to the hardware layer to start the camera.

[0179] S204. Start the camera at the hardware level.

[0180] In some implementations, the hardware layer can start the camera after receiving a request from the HAL layer to start the camera.

[0181] S205. The hardware layer returns information indicating that the camera startup is complete.

[0182] In some implementations, after the hardware layer starts the camera, it can return information about the camera startup being completed to application B, so that application B can perform subsequent steps.

[0183] S206. Application B starts the preview function.

[0184] In some implementations, after receiving the information that the camera has been launched, application B can activate the preview function. The preview function refers to presenting the user with the real-time captured image after the camera application is opened. After application B activates the preview function, the electronic device can begin triggering the acquisition of the raw image. The electronic device triggering the acquisition of the raw image may include steps S207-S209.

[0185] S207. Application B calls the startPreview interface to send a request to the HAL layer to acquire the original image.

[0186] S208.HAL layer configuration parameters and initialization process.

[0187] In some implementations, after receiving a request from application B to acquire the raw image, the HAL layer can execute S208. Specifically, S208 may include the HAL layer powering on the camera, configuring parameters, and allocating streaming.

[0188] Powering on the camera refers to providing power to the camera so that it is in working condition. For example, the HAL layer can call the corresponding power management interface to control the camera's power switch.

[0189] Configuration parameters refer to setting various camera parameters for the camera. For example, the HAL layer passes camera parameters set by the upper-layer application (such as application B) to the camera by calling the interface provided by the camera. In this embodiment, the HAL layer can determine that the current scene is a barcode scanning scene based on the barcode scanning function started by application B, and then configure various camera parameters in the barcode scanning scene.

[0190] Stream configuration can refer to configuring the transmission path and format of data to ensure that data can be correctly transmitted from the upper application layer and framework layer (such as the camera frame) to the camera and from the camera to the upper application for processing. For example, the HAL layer can configure the attributes of the data stream, such as data format, buffer size, and transmission rate, by calling the interface provided by the camera to ensure efficient transmission of the data stream.

[0191] For example, the configuration stream may include a preview stream and / or a data stream. Both the preview stream and the callback stream are data streams. Both the preview stream and the callback stream can include data information of the image captured by the camera. The preview stream is used for real-time display by the application. The callback stream is used by the application for data processing and analysis. Because the preview stream and the callback stream have different functions, the processing methods after the preview stream and the callback stream are returned to the application are different, and therefore the specific data formats of the data information in the preview stream and the callback stream can also be different. For example, the data format of the preview stream can be format33HAL_PIXEL_FORMAT_BLOB, or format34HAL_PIXEL_FORMAT_IMPLEMENTATION_DEFINED. The data format of the callback stream can be format35HAL_PIXEL_FORMAT_YCbCr_420_888.

[0192] In this application embodiment, the target image obtained by the application is used for barcode recognition. That is to say, this application mainly focuses on the callback stream scenario, that is, the original image and target image mentioned in the embodiments of this application belong to the callback stream.

[0193] The S209.HAL layer sends a request to the hardware layer to acquire the raw image.

[0194] In some implementations, the HAL layer can request raw images continuously acquired by the hardware layer after executing S208.

[0195] In some implementations, after receiving a request to acquire raw images, the hardware layer can periodically and continuously acquire raw images, and the HAL layer can also periodically generate target images and determine whether camera parameters need to be adjusted. Specifically, the HAL layer can determine whether camera parameters need to be adjusted based on the current scene (e.g., a barcode scanning scenario).

[0196] For example, after the hardware layer receives a request to acquire the raw image, step group SG21 (S210a-S213a) can be executed.

[0197] S210a. Hardware layer acquires the original image m1.

[0198] S211a. The hardware layer sends the raw image m1 to the HAL layer.

[0199] The S212a.HAL layer generates the target image M1 and determines whether camera parameters need to be adjusted.

[0200] In some implementations, after receiving the original image m1, the HAL layer can generate the target image M1 and determine whether camera parameters need to be adjusted.

[0201] Specifically, after acquiring the original image m1, the HAL layer can generate the target image M1 based on the original image m1. Simultaneously, after acquiring the original image m1, the HAL layer can determine whether camera parameters need adjustment based on the original image m1. In other words, the generation of the target image M1 and the determination of whether camera parameters need adjustment are performed in parallel by the HAL layer.

[0202] In one possible implementation, the parallel execution described above can include the following scenarios:

[0203] Scenario 1: The HAL layer simultaneously generates the target image M1 and determines whether camera parameters need adjustment. For example... Figure 6A As shown, the HAL layer starts generating the target image M1 and determining whether camera parameters need adjustment at the same time. There is no limitation on when the generation of the target image M1 is completed and when the determination of whether parameter adjustment is needed is completed; it depends on the execution efficiency of the HAL layer. For example, they can be completed simultaneously or one after the other.

[0204] Case 2: After the HAL layer begins generating the target image M1, but before completing the generation of the target image M1, it begins to determine whether camera parameters need to be adjusted. For example... Figure 6BAs shown, the time when the generation of the target image M1 begins is before the time when the determination of whether camera parameters need to be adjusted begins, and there is an interval of T1. The duration of T1 is less than the time required to generate the target image M1. In other words, the generation of the target image M1 and the determination of whether camera parameters need to be adjusted overlap in execution time.

[0205] Case 3: After the HAL layer begins to determine whether camera parameters need adjustment, it starts generating the target image M1 before completing this determination. For example... Figure 6C As shown, the start time is before the start time of generating the target image M1, and is separated by a time interval T1. The time interval T1 is less than the time required to determine whether camera parameters need to be adjusted. In other words, the generation of the target image M1 and the determination of whether camera parameters need to be adjusted overlap in execution time.

[0206] For example, Figure 5 The HAL layer shown simultaneously generates the target image M1 and determines whether camera parameters need to be adjusted. In this implementation, Figure 5 Case 1 shown is for illustrative purposes only and is not intended to be limiting.

[0207] In one possible implementation, the HAL layer can determine whether camera parameters need to be adjusted during the generation of the target image M1.

[0208] In this embodiment, the HAL layer can determine whether camera parameters need to be adjusted based on the pre-screening module, barcode information detection module, and camera behavior judgment module. If it is determined that camera parameters need to be adjusted, the target camera parameters can be further obtained, and the target camera parameters and corresponding adjustment instructions can be issued.

[0209] In the above S212a, the HAL determines whether camera parameters need to be adjusted based on the original image m1, which may specifically include the following steps:

[0210] First, the HAL layer inputs the original image m1 into the pre-screening module, which predicts the possible areas where the barcode may exist in the original image m1 and crops out these areas from the original image m1 to obtain the cropped image m1-1.

[0211] The HAL layer then inputs the original image m1 and the cropped image m1-1 into the barcode information detection module, which determines the feature information of the barcode in the original image m1. The feature information of the barcode in the original image m1 may include one or more of the following: the position of the barcode in the original image m1, the category of the barcode in the original image m1, the feature points of the barcode in the original image m1, etc. Optionally, before inputting the original image m1 and the cropped image m1-1 into the barcode information detection module, the HAL layer can also perform preprocessing on the original image m1 and the cropped image m1-1, such as grayscale conversion, normalization, etc.

[0212] Finally, the HAL layer can input the feature information of the barcode in the acquired original image m1 to the camera behavior judgment module, which then determines whether camera parameters need to be adjusted based on the feature information of the barcode in the original image m1. If it is determined that camera parameters need to be adjusted, the camera behavior judgment module can further issue corresponding adjustment commands to the hardware layer based on the determined target camera parameters.

[0213] Specifically, the zoom determination module determines whether zooming is needed based on the position and proportion of the barcode in the original image m1, and further determines the target zoom ratio if zooming is required. The 3A determination module determines whether 3A parameters, such as exposure parameters and focus, need to be adjusted based on the feature information of the barcode in the original image m1, and further determines the target 3A parameters, such as target exposure parameters and target focus, if adjustment of 3A parameters is required. If camera parameter adjustment is determined, the camera behavior determination module can also issue corresponding adjustment instructions to the hardware layer based on the determined target zoom ratio and / or target 3A parameters. Optionally, in addition to the above focus and 3A parameters, the camera behavior determination module can also determine whether other camera parameters need to be adjusted, which is not limited in this embodiment.

[0214] For more detailed information on the pre-screening module, barcode information detection module, and camera behavior judgment module mentioned above, please refer to the subsequent introductions, which will not be elaborated here.

[0215] The S213a.HAL layer sends the target image M1 to application B.

[0216] After the target image M1 is generated in the HAL layer, S213a can be executed. Specifically, the HAL layer can send the target image M1 to application B through the camera frame.

[0217] Optionally, if the HAL layer has already obtained the feature information of the barcode in the original image m1 when executing S213a, the HAL layer can send the feature information of the target image M1 and the barcode to the application B to simplify the processing flow of the application B to identify the barcode in the target image M1.

[0218] Understandably, the electronic device generates the target image M1 and determines the camera parameters that need to be adjusted through the above-mentioned steps SG21.

[0219] After completing S210a-S213a above, you can proceed to step group SG22 (S210b-S213b).

[0220] S210b. Hardware layer acquires the original image m2.

[0221] S211b. The hardware layer sends the raw image m2 to the HAL layer.

[0222] The S212b.HAL layer generates the target image M2 and determines whether camera parameters need to be adjusted.

[0223] The S213b.HAL layer sends the target image M2 to application B.

[0224] Similarly, after executing S210b-S213b above, execute step group SG23 (S210c-S213c) again. The specific steps contained in step group SG22 and step group SG23 can be referred to step group SG21 above, and will not be repeated here.

[0225] It is worth noting that, Figure 5 Each step group shown generates one frame of the target image, and each step group also performs a judgment on whether camera parameters need to be adjusted. In other words, the electronic device can generate one frame of the target image after each step group is executed. However, whether the application receives the target image sent by the HAL layer depends on whether the application is currently idle. If the application is idle, it receives the target image; if the application is busy, it discards the target image. For example, application B receives target images M1 and M3, and discards target image M2.

[0226] Step groups SG21 to SG25 described above are for illustrative purposes only and are not limited to specific steps. Figure 5 The three frames shown illustrate how the electronic device can generate multiple target images after starting to acquire the original images. Furthermore, steps SG21-SG25 are executed by the hardware layer and HAL layer and do not affect the execution of application B within the corresponding time period.

[0227] In some implementations, the HAL layer can determine whether camera parameters need to be adjusted. If it is determined that camera parameters need to be adjusted, an adjustment command is sent to the hardware layer; if it is determined that camera parameters do not need to be adjusted, no adjustment command needs to be sent to the hardware layer.

[0228] The following S214-S215 can be the case where the HAL layer determines that the camera parameters need to be adjusted.

[0229] The S214.HAL layer sends adjustment instructions to the hardware layer.

[0230] In some implementations, after the HAL layer determines whether camera parameters need to be adjusted based on the original image m1, if the HAL layer determines that camera parameters need to be adjusted, it then determines the target camera parameters and executes S214. For example, the target camera parameters include the target zoom ratio and / or the target 3A parameters, and the adjustment instructions include zoom instructions and / or 3A adjustment instructions.

[0231] In one possible implementation, such as Figure 5 As shown, S214 and S213a can be executed simultaneously. In this embodiment, S214 and S213a may also not be executed simultaneously, for example, S214 may be executed after S213a, depending on the specific circumstances of the execution of S212a described above.

[0232] S215. Adjust camera parameters at the hardware level.

[0233] After receiving adjustment instructions from the HAL layer, the hardware layer can adjust camera parameters based on these instructions. For example, when the adjustment instruction includes a zoom instruction, the hardware layer can adjust the camera's zoom ratio to the target zoom ratio. For example, the hardware layer can also change the lens used based on the target zoom ratio, such as replacing it with a telephoto lens. When the adjustment instruction includes a 3A adjustment instruction, the hardware layer can set the camera's 3A parameters to the target 3A parameters.

[0234] For example, such as Figure 5 As shown, the hardware layer completes S215 before executing S210c. That is, the hardware layer acquires the original image m3 after adjusting the camera parameters. The remaining original images m1 and m2 were acquired before adjusting the camera parameters.

[0235] In some implementations, after receiving the target image sent by the HAL layer, application B can process the target image and identify the barcode in the target image to obtain barcode information.

[0236] S216a. Apply B to identify the barcode in the target image M1.

[0237] When the HAL layer sends the target image M1 to application B, application B can process the target image M1 since it is in an idle state. Therefore, application B receives the target image M1.

[0238] After receiving the target image M1, application B can identify the barcode in the target image M1. For details, please refer to the above introduction regarding application B's identification of barcodes in target images; further details will not be elaborated upon here.

[0239] In some implementations, after executing S216a, application B fails to recognize the barcode. The failure of application B to recognize the barcode in the target image M1 may specifically include: failure to detect the position of the barcode in the target image M1, and failure to parse the barcode in the target image M1. For examples, please refer to the above. Figure 3A and Figure 3B The explanation of barcode recognition failure will not be elaborated here.

[0240] Optionally, after executing S216a, if parsing the barcode in the target image M1 fails, application B can issue a zoom command.

[0241] While application B is recognizing the barcode of target image M1, the HAL layer sends target image M2 to application B. However, because application B is busy at this time, it cannot process target image M2, and therefore, application B discards target image M2.

[0242] In one possible implementation, generating the target image M1 based on the HAL layer and determining whether camera parameters need to be adjusted are performed in parallel. S215 and S216a can also be performed in parallel. That is, the hardware layer does not need to wait for application B to determine whether camera parameters need to be adjusted before adjusting the camera parameters.

[0243] S216b. Application B identifies the barcode in target image M3.

[0244] When the HAL layer sends the target image M3 to application B, application B is in an idle state and can process the target image M3. Therefore, application B receives the target image M3.

[0245] S217. After success, application B obtains the barcode information.

[0246] After application B successfully recognizes the barcode in the target image M3, it can obtain the barcode information.

[0247] Optionally, after obtaining the barcode information, application B can perform related operations. For example, if the barcode information includes a link to a page, the electronic device can navigate to and display the page indicated by the barcode information.

[0248] In the embodiments of this application, Figure 5 The interactive process shown is only an example. The parallel execution of generating the target image M1 and determining the target camera parameters is only an example. It is also possible that the electronic device executes the target image M1 and determines that no camera parameters need to be adjusted in parallel, and then executes the generation of the target image M2 and determines the target camera parameters in parallel. When the electronic device executes the scanning method provided in the embodiments of this application, there may be more situations depending on the actual situation, which are not limited.

[0249] from Figure 5 The timing diagram shown shows that the HAL layer can directly determine whether camera parameters need to be adjusted based on the original image, without waiting for application B to fail to recognize the barcode in the target image before determining whether camera parameters need to be adjusted. In this way, the hardware layer only needs to wait a short period of time to complete the adjustment of camera parameters, and then the barcode can be recognized successfully.

[0250] Specifically, the HAL layer determines whether camera parameters need adjustment based on the original image of frame t and then issues an adjustment command. At this time (i.e., the moment the adjustment command is issued), the application layer attempts to recognize the barcode in the target image of frame t, but fails. After receiving the adjustment command, the hardware layer adjusts the camera parameters. Since adjusting camera parameters also takes some time, after the adjustment is complete, the hardware layer can send the original image with adjusted camera parameters to the HAL layer, which then sends the target image with adjusted camera parameters to application B. At this time (i.e., the moment application B receives the target image with adjusted camera parameters), it is already frame t+n. Here, n can be the ratio of the hardware layer's adjustment time to the duration of a single frame. In other words, when application B fails to recognize the barcode in the target image of frame t, it only needs to wait until it obtains the image of frame t+n to successfully recognize the barcode. The overall cycle is short, resulting in a better user scanning experience.

[0251] The following is an introduction Figure 5 The method flow shown is relative to the above. Figure 2 The illustrated method flow yields more beneficial results. For example, combining... Figure 7A and Figure 7B Let's take a look at an example where the duration of a single frame is 30ms. Figure 7A and Figure 7B The times shown on the timeline represent the times when the target image was generated. For example, "t" on the timeline indicates that the target image of frame t was generated, "t+1" on the timeline indicates that the target image of frame t was generated, and so on.

[0252] Figure 7A An example shows the execution of the above. Figure 2 The diagram shows the timing sequence of the scanning method.

[0253] refer to Figure 7AApplication A receives the target image frame t and begins to recognize the barcode in it. After 50ms, application A fails to recognize the barcode in frame t and issues a zoom command. Within 50ms of application A recognizing the barcode in frame t, application A discards frame t+1 of the target image. The hardware layer completes the zoom process after receiving the zoom command, taking 25ms. Within 25ms of the hardware layer performing the zoom process, application A receives the target image frame t+2 and begins to recognize it. After 50ms, application A fails to recognize the barcode in frame t+2. Within 50ms of application A recognizing the barcode in frame t+2, application A discards frame t+3 of the target image. After completing the zoom process, the hardware layer generates the original image frame t+4. Application A receives the target image frame t+4 and begins to recognize the barcode in it. After 50ms, application A successfully recognizes the barcode in frame t+4.

[0254] In other words, Figure 7A In the timing diagram shown, after failing to recognize the barcode in the target image of frame t, the fastest time to successfully recognize the barcode in the target image of frame t+4 is, for example, in Figure 7A In this scenario, application A fails to recognize the barcode twice before it has a chance of successfully recognizing it on the third attempt. For example, in... Figure 7A In this process, starting from the beginning of recognizing the barcode in the target image of frame t, it takes at least 160ms for the barcode to be successfully recognized.

[0255] Figure 7B An example shows the execution of the above. Figure 5 The diagram shows the timing sequence of the scanning method.

[0256] refer to Figure 7BThe HAL layer generates the t-th frame target image in parallel and determines the camera parameters that need adjustment, for example, through the HAL layer's pre-screening module, barcode information detection module, and camera behavior judgment module. Application B receives the t-th frame target image sent by the HAL layer and begins to recognize the barcode in the t-th frame target image. After 50ms, application B fails to recognize the barcode in the t-th frame target image. Within 50ms of application B recognizing the barcode in the t-th frame target image, application B discards the (t+1)-th frame target image. The hardware layer receives the adjustment command sent by the HAL layer and begins to adjust the camera parameters. After 25ms, the hardware layer completes the camera parameter adjustment. The moment application B begins recognizing entries in the t-th frame target image is the same moment the hardware layer begins adjusting the camera parameters. After completing the camera parameter adjustment, the hardware layer generates the (t+2)-th frame original image. Application B receives the (t+2)-th frame target image and begins to recognize the barcode in the (t+2)-th frame target image. After 50ms, application B successfully recognizes the barcode in the (t+2)-th frame target image.

[0257] In other words, Figure 7B In the timing diagram shown, after failing to recognize the barcode in the target image of frame t, the fastest time to successfully recognize the barcode in the target image of frame t+2 is at least. For example, in Figure 7B In this scenario, if application B fails to recognize a barcode once, it may succeed on the second attempt. For example, in... Figure 7B In this process, starting from the beginning of recognizing the barcode in the target image of frame t, it takes at least 100ms for the barcode to be successfully recognized.

[0258] Understandably, if the barcode in the target image obtained after adjusting the parameters is successfully recognized, Figure 7B The scanning process shown can be compared to Figure 7A The scanning process shown indicates that the barcode was successfully recognized in X frames, where X can be any of the smallest integers greater than or equal to the ratio of the application's barcode recognition time to the duration of a single frame. For example, if the application's barcode recognition time is 50ms and the single frame duration is 30ms, the smallest integer greater than the ratio of 50 to 30 is 2, so X is 2.

[0259] The above Figure 7A and Figure 7B The specific processing times for barcode recognition and zoom processing mentioned in the document, such as 50ms and 25ms, are only illustrative examples. In the embodiments of this application, considering various factors such as the performance of electronic devices, the specific processing times for barcode recognition and zoom processing can also be other times, and there is no limitation on them.

[0260] In combination with the above Figures 2 to 7B As can be seen from the illustrated embodiments: Figure 2The scanning method shown requires application A to fail to recognize the barcode in a certain frame of the target image before issuing a zoom command to enable the hardware layer to perform zoom processing. This not only causes application A to discard multiple frames of the target image during recognition, wasting resources, but also requires the hardware layer to wait a considerable amount of time to adjust camera parameters. Application A can only successfully recognize the barcode on the adjusted target image, resulting in excessively long recognition times and a low success rate.

[0261] Figure 5 The scanning method shown does not require waiting for application B to fail to recognize the barcode in the target image before determining whether camera parameters need to be adjusted. The HAL layer can directly determine whether camera parameters need to be adjusted based on the original image. In this way, the hardware layer only needs to wait a short period of time to complete the adjustment of camera parameters. Application B can then successfully recognize the barcode in the target image after the camera parameters have been adjusted. The barcode recognition time is short and the recognition success rate is high.

[0262] The following is combined with Figure 5 This application describes the barcode location detection module, pre-screening module, and camera behavior judgment module provided in its embodiments.

[0263] In some implementations, the barcode information detection module is used to determine the feature information of the barcode in the original image. The barcode feature information includes one or more of the following: barcode position, category, feature points, etc. The barcode information detection module includes modules such as a barcode position detection module and a barcode feature point detection module.

[0264] The barcode location detection module is used to detect the original image and obtain the location and category of the barcode in the original image.

[0265] In the embodiments of this application, the position of the barcode can be represented by various methods or algorithms, and there is no limitation on this. For example, the position of the barcode can be represented by a bounding box. For instance, the position of the barcode can be represented by the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2). A bounding box is an algorithm for finding the optimal bounding space of a discrete set of points. The basic idea is to use a geometric object with a slightly larger volume and simpler characteristics to approximate a complex geometric object.

[0266] In this embodiment, the barcode location detection module can use various specific detection methods to obtain the barcode location from the original image, without limitation. Methods for detecting barcode location include traditional methods and methods based on neural network models. For example, the traditional method refers to determining the barcode location by identifying its edges. The neural network-based method refers to automatically learning complex features in the original image through a neural network model to determine the barcode location. For example, the barcode location detection module can determine the barcode location in the original image using a neural network model with a Cross-Stage Partial Path Aggregation Network (CSP-PAN) network structure.

[0267] Figure 8 An exemplary schematic diagram of a neural network model based on the CSP-PAN structure provided in an embodiment of this application is shown.

[0268] refer to Figure 8 The neural network model used to determine the location of a barcode can include a backbone network, a CSP-PAN structure, and a head. The backbone is the network in the neural network model used to extract the main features. The backbone is typically composed of a pre-trained convolutional neural network, such as a residual network (ResNet) or a visual geometry group network (VGG). The CSP-PAN structure is a feature fusion module in the field of object detection, an improvement on PANet (Path Aggregation Network). The CSP module is a feature processing concept that divides the feature map into two parts: one part undergoes a convolution operation, and the other part is concatenated with the result of the convolution operation. This reduces computational cost and improves feature utilization. The PAN structure adds a top-down feature structure to the FPN, achieving more comprehensive feature fusion. The head, as the output of the neural network model, is responsible for making specific task predictions based on the fused feature information, such as object detection, image segmentation, etc. For example, in an object detection task, the head includes a classification head and a regression head, used to predict the object's category and bounding box location, respectively.

[0269] For example, the original image is input into the Backbone of the neural network model. The Backbone extracts features at different levels and scales from the original image, generating feature information of the original image, and inputs this feature information into the CSP-PAN structure. The CSP-PAN structure receives the feature information of the original image input from the Backbone, fuses the features through different paths to obtain the feature information of the fused original image, and inputs the feature information of the fused original image into the subsequent network Head. The Head receives the feature information of the fused original image sent by the CSP-PAN structure, determines the position of the barcode in the original image through a regression head, and optionally, the Head can also determine the category of the barcode in the original image through a classification head.

[0270] In the embodiments of this application, the method of obtaining the position and category of the barcode using a neural network model based on the CSP-PAN structure is only illustrative. Other neural network models can also be used, and there is no limitation on this.

[0271] In some implementations, an original image may contain one barcode, multiple barcodes, or other similar barcode-like graphics. The barcode location detection module can detect multiple regions that may contain barcodes. Furthermore, in this case, the barcode location detection module can also determine the confidence level of the detected regions that may contain barcodes. The barcode location detection module has a confidence threshold; when the confidence level of a region that may contain a barcode exceeds this threshold, the confidence level is considered valid. Further, among the detected regions that may contain barcodes, the barcodes contained in regions where the confidence level exceeds the confidence threshold are determined to be valid barcodes. And, the barcode contained in the region with the highest confidence level is determined to be the barcode in the original image.

[0272] The aforementioned regions that may contain barcodes can be understood as follows: the regions in the original image that the barcode location detection module considers to potentially contain barcodes are not necessarily regions in the original image that actually contain barcodes.

[0273] Optionally, the barcode location detection module can also determine the type of the barcode. Specifically, the barcode location detection module can determine whether the barcode contained in the original image is a one-dimensional barcode or a two-dimensional barcode. Further, after determining that the barcode is a two-dimensional barcode, it determines the specific type of the two-dimensional barcode. For example, the barcode location detection module can calculate the confidence level of a detected barcode belonging to various barcode types, and determine the barcode type with the highest confidence level as the type of that barcode.

[0274] The barcode feature point detection module is used to detect the barcode's feature points in the original image. Specifically, after the barcode position detection module processes the original image, it can obtain the barcode's position within the original image. The barcode information detection module crops the area containing the barcode in the original image, obtaining a cropped original image. This cropped original image is then input into the barcode feature point detection module to obtain the coordinates of the barcode's feature points.

[0275] In the embodiments of this application, the feature points of a barcode can specifically refer to one or more points in the barcode. For example, taking a QR code as an example, the feature points can refer to the corner points of the QR code; the feature points can also refer to the calibration points of the QR code; the feature points can also refer to the corner points of the positioning pattern in the QR code.

[0276] Figure 9A An exemplary schematic diagram of the morphological structure of the QR Code provided in the embodiments of this application is shown.

[0277] refer to Figure 9A , Figure 9A (a) shows the morphological structure of a QR code. A QR code includes three positioning graphics in the upper left, upper right, and lower left corners, as well as a calibration graphic in the lower right corner.

[0278] Figure 9A (b) exemplarily illustrates a feature point of a QR code. The feature points of a QR code refer to its corner points, namely the bottom left, top left, top right, and bottom right corners, for example, Figure 9A Points 1, 2, 3 and 4 are shown in (b).

[0279] Figure 9A (c) exemplarily illustrates another type of QR code feature point. A QR code feature point refers to the calibration point of the QR code, i.e., the center point of the calibration graphic, for example, Figure 9A Point 1 is shown in (c).

[0280] Figure 9A Example (d) illustrates yet another type of QR code feature point. A QR code feature point refers to the corner points of the seek graphic within the QR code, specifically the bottom left, top left, top right, and bottom right corners of the seek graphic. For example, Figure 9A Points 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 and 12 are shown in (d).

[0281] In the embodiments of this application, Figure 9AThe feature points shown in the barcode are for illustrative purposes only and may refer to other points in the barcode; there is no limitation on this.

[0282] In this embodiment, the barcode location detection module can use various specific detection methods to obtain barcode feature points from the original image, without limitation. Methods for detecting barcode feature points include traditional methods and methods based on neural network models. The aforementioned neural network-based method refers to automatically learning complex features in the original image through a neural network model to determine the barcode feature points. For example, the barcode feature point detection module can obtain heatmaps of the four corner points of the QR code and determine the four points with the strongest response and a threshold value as the barcode feature points.

[0283] In one possible implementation, the feature points of the barcode acquired by the barcode position detection module can be as follows: Figure 9A (b) Figure 9A (c) and Figure 9A The feature point shown in (d) can also be as follows: Figure 9B The feature points shown in (a) are due to the small angle between the physical QR code plane and the electronic device's camera. This means that when the user scans the code, the electronic device is not directly facing the physical barcode; for example, the top or bottom of the electronic device is closer to the barcode, or the left or right side is closer. In this scanning scenario, the original image captured by the electronic device will be distorted, for example, causing trapezoidal or barrel distortion of the barcode during imaging. Consequently, the feature points of the barcode obtained by the barcode position detection module based on the original image will be as shown... Figure 9B As shown in (a). Figure 9B The feature points in (a) are the focal points of the barcode, namely the bottom left, top left, top right, and bottom right corners of the barcode, for example, Figure 9B Points 1, 2, 3, and 4 are shown in (a). It can be seen that: Figure 9B The barcode formed by the feature points in (a) is distorted.

[0284] Electronic devices acquire such Figure 9B After identifying the feature points of the barcode shown in (a), these feature points can be sent to the application side for correction processing to restore the barcode feature points from the original image captured without distortion, for example... Figure 9B The feature points shown in (b) are as follows. Specifically, correction processing can typically be achieved through distortion correction algorithms, image enhancement techniques, etc. In the embodiments of this application, the above-mentioned correction processing is not limited.

[0285] Figure 10 An exemplary schematic diagram of the process for obtaining feature points of a barcode provided in an embodiment of this application is shown.

[0286] refer to Figure 10 The process begins by converting the cropped original image into a digital form, such as a pixel matrix, which is then input into the encoder. The encoder processes the cropped original image and extracts its features, such as point feature maps containing features from the cropped original image. The encoder is typically a convolutional neural network. These point feature maps are then fed into a linear layer. The linear layer, acting as a neural network layer, converts the point feature maps into a one-dimensional heatmap using matrix multiplication. Each value in the heatmap represents the feature intensity or response intensity at the corresponding location. Peak positions in each heatmap may correspond to feature points of the barcode in the cropped original image. Finally, by analyzing the 1D heatmap, the four points with the strongest responses exceeding a threshold are identified; these are the feature points of the barcode.

[0287] In the embodiments of this application, the above Figure 10 The method of obtaining barcode feature points shown is only an example. Other methods can also be used to obtain barcode feature points, and there are no limitations on this.

[0288] In this embodiment, the barcode information detection module is deployed in the HAL layer. The HAL layer can obtain barcode feature information such as the barcode's location, category, and feature points. The HAL layer can determine whether camera parameters need to be adjusted based on the above feature information, and can also send the above feature information to the corresponding application at the same time as sending the original image to the corresponding application, thereby reducing application processing time and improving application processing efficiency.

[0289] The pre-screening module predicts the possible locations of barcodes in the original image. Specifically, before the barcode detection module executes its function, the pre-screening module crops the areas in the original image where barcodes might be present, resulting in a cropped original image. This is because if the original image is directly input into the barcode detection module, and the barcode occupies less than a certain percentage (e.g., 20%), the module may be unable to determine its location. This would give the barcode detection module the illusion that the original image does not contain a barcode, thus affecting subsequent adjustments to camera parameters. Therefore, the pre-screening module can also be understood as increasing the proportion of the barcode in the image, thereby improving the success rate of the barcode detection module in acquiring barcode feature information.

[0290] In this embodiment, the pre-screening module can use various specific methods to crop the areas where barcodes may exist in the original image to obtain a cropped original image, and there is no limitation on these methods. Methods for obtaining the cropped image include manually designed anchor boxes and machine learning-based methods, such as machine learning-based salient region detection algorithms.

[0291] The following describes the method flow of the pre-screening module provided in the embodiments of this application, which is based on obtaining the clipping region using a priori boxes.

[0292] Step a: Set the prior box.

[0293] Prior boxes (or anchor boxes) are used to improve the accuracy of target localization and classification. A prior box is a set of rectangles of different sizes and proportions pre-defined on an image, used to match and predict possible target objects in the image. There can be one or more prior boxes. The size, dimensions, and position of the prior boxes can be generated using clustering algorithms (such as K-means) or set manually; there are no restrictions on this.

[0294] Figure 11A An exemplary diagram of setting a priori box is shown in an embodiment of this application.

[0295] refer to Figure 11A The pre-screening module sets five prior boxes for the original image: prior box 1, prior box 2, prior box 3, prior box 4, and prior box 5. For example, Figure 11A The gray rectangle in the image can be considered a barcode in the original image.

[0296] In the embodiments of this application, the above Figure 11A The prior bounding boxes provided are for illustrative purposes only. The pre-screening module can also set prior bounding boxes with other data and sizes, without limitation. For example, based on the assumption that users tend to center the barcode on the screen when scanning, the pre-screening module can set only one prior bounding box located in the center of the original image, with a size of 1 / 6 of the original image.

[0297] Step b-1: For the first frame of the original image, the pre-screening module determines the prior box located at the center of the original image from the multiple prior boxes set as the final selected prior box.

[0298] This is because the original image is the first one obtained, and there are no other original images available for reference. Therefore, the center area of ​​the original image is assumed to be the area where the barcode is located.

[0299] Step b-2: For the original image of frame t, the pre-screening module determines the final selected prior box from multiple prior boxes based on the barcode position detection result of the original image of frame t-1 output by the barcode information detection module.

[0300] Understandably, the barcode information detection module described above can identify multiple regions in the original image that may contain barcodes during execution. The module can also determine the confidence level of these multiple regions. Barcodes contained within regions with a confidence level higher than a confidence threshold are identified as valid barcodes by the barcode information detection module. In other words, the barcode location detection result includes the confidence levels of multiple regions in the (t-1)th frame of the original image that may contain barcodes (including the confidence levels of regions containing valid barcodes and regions containing invalid barcodes).

[0301] In step b-2, the pre-screening module obtains multiple regions in the (t-1)th frame image that may contain barcodes, along with the confidence levels of these regions, and assigns them to various prior boxes. For example, the assignment can be determined by calculating the cost matrix between the multiple regions and the prior boxes, thus transforming the assignment of these regions to the prior boxes into an optimal transmission problem.

[0302] Figure 11B An exemplary diagram illustrates the possible locations of barcodes in the original image of frame t-1.

[0303] refer to Figure 11B The pre-screening module sets two prior boxes for the original image of frame t-1, namely prior box 1 and prior box 2. Figure 11B The gray rectangles in the diagram represent regions within the original image of frame t-1 that may contain barcodes, detected by the barcode detection information module. For example, regions that may contain barcodes include box1, box2, box3, box4, box5, and box6. Box1, box2, and box3 fall within the prior bounding boxes, while box4, box5, and box6 do not. The pre-screening module can assign boxes1, box2, and box3, which are located within the prior bounding boxes, to prior bounding boxes 1 and 2, respectively, i.e., predict whether boxes1, box2, and box3 are most likely located in prior bounding box 1 or prior bounding box 2. For example, the cost matrix of boxes1, box2, and box3 relative to prior bounding boxes 1 and 2 can be calculated using IOU Loss and L2 Loss.

[0304] Intersection over Union (IOU) loss is a common loss function used in object detection tasks to measure the overlap between predicted bounding boxes (such as regions that may contain barcodes) and ground truth bounding boxes (such as prior boxes). Mean Squared Error (L2) loss measures the model's prediction error by calculating the average of the squared differences between predicted and true values. For example, L2 loss is used to measure the distance between regions that may contain barcodes and the center point of prior boxes.

[0305] cost = w dis L dis +w iou L iou (1)

[0306] Formula (1) is a weighted loss function used to calculate the cost of regions that may contain barcodes relative to prior boxes. The pre-screening module assigns regions that may contain barcodes to prior boxes with lower costs. dis and L iou These represent different loss terms. dis L represents dis The weight, w iou L represents iou The weights are used to adjust L dis and L iou Specific gravity. L dis This is used to measure the distance between the area that may contain a barcode and the center point of the prior bounding box, specifically calculated using L2 Loss. iou This is used to measure the degree of overlap between the area that may contain a barcode and the prior bounding box, which can be specifically calculated using IOU Loss.

[0307]

[0308] Formula (2) is used to calculate L. dis . This represents the relevant data obtained by the pre-screening module regarding areas that may contain barcodes, such as the coordinates, width, and height of the area. N represents the number of categories of relevant data for areas that may contain barcodes. i This represents the relevant data of the prior bounding box, such as its coordinates, width, and height. L dis The smaller the value, the smaller the distance between the area that may contain the barcode and the center point of the prior box.

[0309]

[0310] L iou=1-IOU(4)

[0311] Formulas (3) and (4) are used to calculate L. iou AreaofOverlap represents the intersection area of ​​the region that may contain the barcode and the prior frame; that is, the area of ​​the overlapping portion of the region that may contain the barcode and the prior frame. AreaofUnion represents the union area of ​​the region that may contain the barcode and the prior frame; that is, the total area covered by the region that may contain the barcode and the prior frame, including the overlap. The value of IOU ranges from 0 to 1; a higher IOU value indicates a greater degree of overlap between the region that may contain the barcode and the prior frame. When IOU is 1, it means that the region that may contain the barcode and the prior frame completely overlap; when IOU is 0, it means that the region that may contain the barcode and the prior frame do not overlap. iou The value of L ranges from 0 to 1. iou The smaller the value, the higher the degree of overlap between the area that may contain the barcode and the prior box.

[0312] cost box1 box2 box3 Prior box 1 0.7 0.8 1.4 Prior box 2 0.1 0.4 0.3

[0313] Table 1

[0314] Table 1 provides an example. Figure 11B The costs of boxes 1, 2, and 3 with prior boxes 1 and 2 are calculated using formula (1) above. Specifically, the cost of box 1 with prior box 1 is 0.7, and the cost of box 1 with prior box 2 is 0.1. The pre-screening module can assign box 1 to the lower-cost prior box 2. The cost of box 2 with prior box 1 is 0.8, and the cost of box 2 with prior box 2 is 0.4. The pre-screening module can assign box 2 to the lower-cost prior box 2. The cost of box 3 with prior box 1 is 1.4, and the cost of box 3 with prior box 2 is 0.3. The pre-screening module can assign box 3 to the lower-cost prior box 2.

[0315] In the embodiments of this application, the algorithms or methods used to calculate the cost matrix of multiple possible barcodes and prior frames are merely illustrative examples. Other methods and algorithms can also be used for calculation, and no limitation is made thereto.

[0316] After assigning regions that may contain barcodes to various prior boxes, a score can be calculated for each prior box. The score of a prior box is the sum of the confidence levels of the regions that may contain barcodes assigned to that prior box:

[0317]

[0318] Formula (5) is used to calculate the score of the prior box. n This represents the confidence level of a region that may contain a barcode assigned to the prior box. K represents the number of regions that may contain a barcode assigned to the prior box. For example, as shown in Table 1 above: no region that may contain a barcode was assigned to prior box 1, and boxes 1, 2, and 3 were assigned to prior box 2. According to formula (5), the score of prior box 1 is determined to be 0, and the score of prior box 2 is the sum of the confidence levels of boxes 1, 2, and 3. The score of prior box 2 is higher than that of prior box 1, that is, the final selected prior box is prior box 2.

[0319] In this embodiment of the application, the method of determining the final selected prior box from the multiple prior boxes shown above is only an illustrative example. In step b-2, the final selected prior box can also be determined in many other ways, which are not limited.

[0320] Step c: In the original image, crop the region containing the final selected prior box to obtain the cropped original image.

[0321] The pre-screening module can execute step c after performing step b-1 or step b-2 above.

[0322] Figure 11C An exemplary diagram illustrates the input and output information of the barcode information detection module provided in an embodiment of this application.

[0323] refer to Figure 11C The original image and the cropped original image can be input into the barcode information detection module. The barcode information detection module can process the original and cropped original images to obtain barcode feature information, such as barcode location, feature points, and category. Optionally, before inputting them into the barcode information detection module, the original and cropped images can undergo preprocessing such as grayscale conversion, normalization, and resolution reduction. Grayscale conversion refers to the process of converting a color image to a grayscale image, which can reduce color information in the original image, simplify the data image, and improve computational efficiency. Specifically, grayscale conversion can be performed using the average method or the weighted average method. Normalization refers to adjusting the image data to a specific range, typically adjusting the image pixel values ​​from [-255, 255] to [-1, 1]. Reducing the resolution can reduce file size and transmission bandwidth. For example, reducing the image resolution from 1080p to 300 pixels * 300 pixels. This reduces the computational load of the barcode information detection module and speeds up the acquisition of barcode feature information from the original image.

[0324] In this embodiment, the pre-screening module is deployed in the HAL layer. The HAL layer can crop the area where the barcode may exist in the current frame's original image based on the barcode position detection result in the previous frame's original image, resulting in a cropped original image. This increases the proportion of the barcode in the image, improves the success rate of the barcode information detection module in obtaining the barcode's feature information, and increases the maximum recognition distance for scanning.

[0325] In some implementations, the camera behavior determination module is used to determine whether camera parameters need to be adjusted and whether corresponding adjustment commands need to be sent to the hardware layer. The camera behavior determination module includes modules such as a zoom determination module and a 3A determination module.

[0326] The zoom determination module is used to determine whether zooming is needed and to determine the target zoom ratio. Specifically, the zoom determination module can determine whether zooming is needed based on the position and proportion of the barcode in the original image m1. For example, if the proportion of the barcode in the original image is less than 20%, and / or the barcode is incomplete in the original image, then zooming is needed; the target zoom ratio is the zoom ratio that can increase the proportion of the barcode in the original image to 20%. In this embodiment, the proportion is not limited to 20%, and other proportions can be used to determine whether zooming is needed; this is not limited.

[0327] The 3A judgment module is used to determine whether the 3A parameters need to be adjusted and to determine the target 3A parameters.

[0328] The 3A judgment module determines the target focus parameter as follows: When the barcode information detection module detects a barcode in the original image for the previous N consecutive frames, it sets the center point coordinates of the detected barcode in the current frame's original image as the camera's focus. Specifically, the focuscountor counter is set to 0. If the IOU value between the barcode in the current frame's original image and the barcode in the previous frame's original image is greater than a preset value, such as 0.5, the focuscountor counter is incremented by 1. At this point, the IOU value between the barcode in the current frame's original image and the barcode in the previous frame's original image refers to the ratio of the intersection area of ​​the barcode in the current frame's original image and the union area of ​​the barcode in the previous frame's original image. When the focuscountor counter exceeds a preset number of times, such as 5 times, the 3A judgment module calls the setFocusPoint interface to set the center point coordinates (x, y) of the barcode in the current frame's original image as the camera's focus, i.e., the target focus parameter.

[0329] In the embodiments of this application, the above-described method for determining the target focus parameters is only an example. Other methods can also be used to determine the target focus parameters, and no limitation is made thereto.

[0330] The 3A judgment module can determine the target exposure parameters by judging the grayscale value of the area where the barcode is located in the original image of the current frame. For example, refer to... Figure 11D ,exist Figure 11D In the original image, the width is *a* and the height is *b*. The region containing the barcode in the original image has a width of *x* and a height of *y*, and the barcode is located within the regions of 1 / 3a to 2 / 3a and 1 / 3b to 2 / 3b of the original image. In this case, calculate... Figure 11D The grayscale value within the medium gray rectangular region D, specifically the grayscale values ​​within the regions 1 / 4x to 3 / 4x and 1 / 4y to 3 / 4y of the barcode's location. Optionally, if the barcode is not located within the regions 1 / 3a to 2 / 3a and 1 / 3b to 2 / 3b of the original image, then the grayscale values ​​within the regions 1 / 3a to 2 / 3a and 1 / 3b to 2 / 3b of the original image are calculated. When the grayscale value in region D is less than or equal to 50, the target exposure parameter is set to 2; when the grayscale value in region D is greater than 50 and less than or equal to 90, the target exposure parameter is set to 1; when the grayscale value in region D is greater than 90 and less than or equal to 140, the target exposure parameter is set to 0; when the grayscale value in region D is greater than 140 and less than or equal to 190, the target exposure parameter is set to -1; when the grayscale value in region D is greater than 190, the target exposure parameter is set to -1. The 3A judgment module calls the corresponding camera interface to adjust the hardware layer exposure parameters to the target exposure parameters.

[0331] In the embodiments of this application, the above-described method for determining the target exposure parameters is only an example. Other methods can also be used to determine the target exposure parameters, and no limitation is made thereto.

[0332] The following describes a method for scanning codes provided in an embodiment of this application.

[0333] Figure 12 An exemplary flowchart of a barcode scanning method provided in an embodiment of this application is illustrated. This method is applied to an electronic device whose HAL layer includes a barcode position detection module, a pre-screening module, and a camera behavior judgment module. The method includes:

[0334] S301. In response to launching the barcode scanning function of the barcode scanning application, the electronic device acquires a first original image using first camera parameters through the hardware layer. The first original image contains a barcode.

[0335] For example, a QR code scanning application can refer to the above. Figure 5 Application B. In the case where the first raw image is the first frame captured after the electronic device starts the camera, the first camera parameters can be the default camera parameters.

[0336] S302, The electronic device generates a first target image based on the first original image through the HAL layer, and executes in parallel the action of determining the parameters of the first target camera based on the first original image through the HAL layer.

[0337] The above parallel execution can be referenced. Figures 6A to 6C The diagram shown is shown in the image.

[0338] In one possible implementation, after executing S302, the electronic device can further transmit the first target image to the barcode scanning application via the HAL layer. Then, the barcode scanning application identifies the barcode in the first target image; wherein, while identifying the barcode in the first target image via the barcode scanning application, the electronic device can also adjust its camera parameters to the parameters of the first target camera via the hardware layer. For example, refer to... Figure 5 The two steps S216a and S215 in the process.

[0339] Specifically, the electronic device responds by generating a first target image based on a first original image through the HAL layer, and then sends the first target image to the barcode scanning application through the HAL layer.

[0340] S303. Electronic devices transmit the parameters of the first target camera to the hardware layer through the HAL layer.

[0341] Specifically, in response to the analysis of the first raw image by the HAL layer, the electronic device transmits the parameters of the first target camera to the hardware layer through the HAL layer. For example, the HAL layer determines, based on the analysis of the first raw image, that the camera parameters need to be adjusted and thus determines the parameters of the first target camera.

[0342] In one possible implementation, S303 may include: the electronic device obtaining first feature information of the barcode in the first original image through the HAL layer. For example, the first feature information can be obtained through a barcode information detection module in the HAL layer. The electronic device then determines the parameters of the first target camera based on the first feature information of the barcode in the first original image through the HAL layer. For example, the first target camera parameters can be obtained through a camera behavior judgment module in the HAL layer.

[0343] In one possible implementation, before obtaining the first feature information of the barcode in the first original image through the HAL layer, the electronic device can further crop the first original image through the HAL layer to obtain a first cropped image, which includes the barcode. The electronic device can obtain the first feature information of the barcode in the first original image based on the first original image and the first cropped image through the HAL layer, for example, referring to the above. Figure 11C The relevant explanations. For example, it can be done through... Figure 5 The pre-screening module shown in the figure crops the original image m1 to obtain the cropped original image.

[0344] Specifically, the electronic device determines a first prior bounding box from a preset prior bounding box based on the feature information of the barcode in the third original image through the HAL layer. The third original image is acquired before the first original image. Then, the electronic device can crop out the region containing the first prior bounding box in the first original image through the HAL layer to obtain a first cropped image. For example, refer to the above introduction about the pre-screening module and the introduction about... Figure 11A and Figure 11B Related information.

[0345] In one possible implementation, the first target camera parameters include a first target zoom ratio. Specifically, S303 may include: the electronic device determining a first proportion of the barcode in the second original image through the HAL layer. Then, the electronic device determines the first target zoom ratio based on the first proportion through the HAL layer. When the position of the electronic device remains unchanged, the proportion of the barcode in the original image acquired using the first target zoom ratio is greater than the first proportion. For example, refer to the above description of the zoom determination module.

[0346] In one possible implementation, the first target camera parameters include first target exposure parameters. Specifically, S303 includes: the electronic device determining the first grayscale value of all or part of the area containing the barcode in the second original image through the HAL layer; and the electronic device then determining the first target exposure parameters based on the first grayscale value through the HAL layer.

[0347] In one possible implementation, the first target camera parameters include first target focus parameters, and S303 specifically includes:

[0348] The electronic device determines the first coordinates of the barcode center in the second original image through the HAL layer. The electronic device then determines the first target focus parameters based on the first coordinates through the HAL layer. For example, refer to the above description of the 3A judgment module and... Figure 11D Related information.

[0349] S304. The electronic device acquires a second raw image using the parameters of the first target camera through the hardware layer. The second raw image contains a barcode.

[0350] For example, the second original image may refer to Figure 5 The original image m3 shown is illustrated. In this embodiment, there is no limitation on whether the first original image and the second original image are two consecutive frames; this is determined by the hardware layer adjusting camera parameters.

[0351] S305. The electronic device generates a second target image based on the second original image through the HAL layer.

[0352] In one possible implementation, the electronic device can also obtain the second feature information of the barcode in the second original image through the HAL layer, wherein the action of obtaining the second feature information of the barcode in the second original image through the HAL layer and the action of generating the second target image based on the second original image through the HAL layer are performed in parallel.

[0353] S306. The electronic device obtains the information carried by the barcode in the second target image through a barcode scanning application.

[0354] In one possible implementation, the information carried by the barcode in the second target image is obtained by a barcode scanning application based on the second feature information.

[0355] The second feature information includes information such as the location, category, and feature points of the barcode in the second target image.

[0356] In one possible implementation, the second feature information includes a first feature point of the barcode in the second original image. This first feature point is in a distorted state, and the electronic device can correct it to obtain a corrected first feature point. The electronic device can also obtain information carried by the barcode in the second target image based on the corrected first feature point. For example, refer to the above. Figure 9B The diagram shown is shown in the image.

[0357] In one possible implementation, the electronic device can also acquire a third raw image using the parameters of the first camera through a hardware layer. The electronic device generates a third target image based on the third raw image through a HAL layer. Then, the electronic device discards the third target image via a barcode scanning application. Here, the acquisition of the third raw image occurs after the acquisition of the first raw image and before the acquisition of the second raw image. For example, the third raw image could refer to… Figure 5 The original image m2 shown, and the third target image can refer to... Figure 5 The target image M2 is shown.

[0358] In one possible implementation, if the first target image is Figure 5 If the target image M2 shown is discarded by the electronic device via a barcode scanning application, then the electronic device will discard the first target image.

[0359] Implement the above Figure 12 The method shown eliminates the need to wait for the application to fail to recognize the barcode in the target image before determining whether camera parameters need to be adjusted. The HAL layer can directly determine whether camera parameters need to be adjusted based on the original image. In this way, the hardware layer only needs to wait a short period of time to complete the adjustment of camera parameters. The application can then successfully recognize the barcode in the target image after the camera parameters have been adjusted. The barcode recognition time is short and the recognition success rate is high.

[0360] Furthermore, the HAL layer's step of determining whether camera parameters need adjustment does not affect the application's acquisition of the target image. In other words, the HAL layer can generate the target image and determine whether camera parameters need adjustment in parallel. This allows the application to display the acquired target image at the correct frame rate and recognize the QR code within the target image.

[0361] Figure 13 A schematic diagram of the structure of the electronic device 100 is shown.

[0362] The following description uses electronic device 100 as an example to illustrate the embodiment. It should be understood that... Figure 13 The electronic device 100 shown is merely an example, and the electronic device 100 may have more than Figure 13 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0363] For example, electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. This application embodiment does not impose any special restrictions on the specific type of electronic device 100.

[0364] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0365] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0366] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0367] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0368] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0369] In this embodiment, processor 110 is configured to, in response to launching the barcode scanning function of a barcode scanning application, acquire a first original image using first camera parameters through a hardware layer. The first original image contains a barcode. Processor 110 is further configured to generate a first target image based on the first original image through a HAL layer, and concurrently execute the action of determining first target camera parameters based on the first original image through the HAL layer. Processor 110 is also configured to transmit the first target camera parameters to the hardware layer through the HAL layer. Processor 110 is further configured to acquire a second original image using the first target camera parameters through a hardware layer. The second original image contains a barcode. Processor 110 is further configured to generate a second target image based on the second original image through a HAL layer. Processor 110 is also configured to obtain information carried by the barcode in the second target image through the barcode scanning application. Specifically, processor 110 is configured to execute any embodiment of the barcode scanning method provided in this embodiment.

[0370] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0371] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0372] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0373] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110.

[0374] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0375] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0376] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0377] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0378] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0379] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-CDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0380] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0381] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0382] In this embodiment of the application, the display screen 194 is used to display the interface of the scanning application, such as the interface for starting the scanning function and the interface after starting the scanning function.

[0383] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0384] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, etc. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0385] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0386] In this embodiment of the application, the camera module, including camera 193, can be used to collect raw image data.

[0387] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0388] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0389] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0390] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0391] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0392] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0393] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0394] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0395] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0396] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0397] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0398] Pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. Capacitive pressure sensors may include at least two parallel plates with conductive materials.

[0399] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 100, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 100 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0400] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0401] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip cover. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. Then, based on the detected opening and closing state of the cover or the flip cover, features such as automatic flip unlocking can be set.

[0402] The 180E accelerometer can detect the magnitude of acceleration of electronic device 100 in various directions (typically three axes). When electronic device 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0403] A distance sensor 180F is used to measure distance. Electronic device 100 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 100 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0404] The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode.

[0405] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also work with the proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0406] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0407] Temperature sensor 180J is used to detect temperature. In some embodiments, electronic device 100 uses the temperature detected by temperature sensor 180J to execute a temperature processing strategy.

[0408] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0409] For example, the 180K touch sensor can detect a user's touch operation, which is used to activate the barcode scanning function, which requires the camera to be activated.

[0410] The bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire vibration signals from the vibrating bone segments of the human vocal cords.

[0411] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0412] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0413] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0414] The SIM card interface 195 is used to connect the SIM card.

[0415] It is understood that, in order to implement the scanning method provided in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0416] This application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the above-described barcode scanning method.

[0417] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0418] This application also provides a computer program product, including a computing program, which, when run on a computer, enables the computer to perform the steps in the various method embodiments described above.

[0419] This application also provides a chip system, which includes a processing circuit interface circuit. The interface circuit receives code instructions and transmits them to the processing circuit. The processing circuit executes the code instructions to enable the chip system to implement the steps of any method embodiment of this application. The chip system can be a single chip or a chip module composed of multiple chips.

[0420] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0421] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0422] For example, the chip system may be a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on chip (SoC), a central processor (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0423] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0424] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0425] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0426] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for scanning codes, characterized in that, The method is applied to an electronic device, which includes: a barcode scanning application, a Hardware Abstraction Layer (HAL) layer, and a hardware layer. The method includes: In response to activating the scanning function of the scanning application, a first original image is acquired using the first camera parameters through the hardware layer, the first original image containing a barcode; The first target image is generated based on the first original image through the HAL layer, and the action of determining the first target camera parameters based on the first feature information of the barcode in the first original image through the HAL layer is executed in parallel. The parameters of the first target camera are transmitted to the hardware layer through the HAL layer. A second raw image is acquired using the parameters of the first target camera through the hardware layer, and the second raw image contains a barcode. The second target image is generated based on the second original image through the HAL layer; The information carried by the barcode in the second target image is obtained through the scanning application.

2. The method according to claim 1, characterized in that, After generating a first target image based on the first original image through the HAL layer, and performing the action of determining the first target camera parameters based on the first original image through the HAL layer in parallel, the method further includes: The first target image is transmitted to the scanning application through the HAL layer; The barcode in the first target image is identified using the scanning application; When the barcode in the first target image is identified by the scanning application, the camera parameters of the electronic device are adjusted to the camera parameters of the first target image through the hardware layer.

3. The method according to claim 2, characterized in that, The step of transmitting the first target image to the scanning application through the HAL layer specifically includes: In response to generating the first target image based on the first original image through the HAL layer, the first target image is sent to the scanning application through the HAL layer; The specific steps of transmitting the first target camera parameters to the hardware layer through the HAL layer include: In response to the analysis of the first raw image through the HAL layer, the first target camera parameters are passed to the hardware layer through the HAL layer.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The second feature information of the barcode in the second original image is obtained through the HAL layer, wherein the action of obtaining the second feature information of the barcode in the second original image through the HAL layer and the action of generating a second target image based on the second original image through the HAL layer are performed in parallel. The information carried by the barcode in the second target image obtained through the scanning application specifically includes: The scanning application obtains the information carried by the barcode in the second target image based on the second feature information.

5. The method according to claim 4, characterized in that, The second feature information includes a first feature point of the barcode in the second original image, where the first feature point is in a distorted state. The information carried by the barcode in the second target image obtained by the scanning application based on the second feature information specifically includes: The first feature point is corrected to obtain the corrected first feature point. The information carried by the barcode in the second target image is obtained based on the first feature point after correction.

6. The method according to any one of claims 1-3, characterized in that, Before obtaining the first feature information of the barcode in the first original image through the HAL layer, the method further includes: The first original image is cropped through the HAL layer to obtain a first cropped image, which includes a barcode. The first feature information of the barcode in the first original image obtained through the HAL layer specifically includes: The first feature information of the barcode in the first original image is obtained through the HAL layer based on the first original image and the first cropped image.

7. The method according to claim 6, characterized in that, The cropping of the first original image through the HAL layer specifically includes: The HAL layer determines the first prior box from the preset prior box based on the feature information of the barcode in the third original image. The third original image is acquired before the first original image. The first cropped image is obtained by cropping the region containing the first prior box in the first original image through the HAL layer.

8. The method according to any one of claims 1-3, characterized in that, The first target camera parameters include a first target zoom ratio. The step of determining the first target camera parameters through the HAL layer based on the first feature information of the barcode in the first original image specifically includes: The first proportion of the barcode in the first original image is determined by the HAL layer; The first target zoom ratio is determined by the HAL layer based on the first proportion. When the position of the electronic device does not change, the proportion of barcode in the original image acquired using the first target zoom ratio is greater than the first proportion.

9. The method according to any one of claims 1-3, characterized in that, The first target camera parameters include first target exposure parameters. The step of determining the first target camera parameters through the HAL layer based on the first feature information of the barcode in the first original image specifically includes: The first grayscale value of all or part of the area where the barcode is located in the first original image is determined by the HAL layer. The first target exposure parameters are determined by the HAL layer based on the first grayscale value.

10. The method according to any one of claims 1-3, characterized in that, The first target camera parameters include a first target focus parameter. The step of determining the first target camera parameters using the HAL layer based on the first feature information of the barcode in the first original image specifically includes: The first coordinates of the barcode center in the first original image are determined by the HAL layer; The HAL layer determines the first target focus parameters based on the first coordinates.

11. The method according to any one of claims 1-3, characterized in that, The method further includes: A fourth raw image is acquired using the first camera parameters through the hardware layer; A fourth target image is generated based on the fourth original image through the HAL layer; The fourth target image is discarded via the scanning application; The fourth original image is acquired after the first original image is acquired and before the second original image is acquired.

12. The method according to any one of claims 1-3, characterized in that, The method further includes: The first target image is discarded by the scanning application.

13. An electronic device, characterized in that, The electronic device includes a memory, a processor, and computer instructions stored in the memory, the processor executing the computer instructions to implement the method of any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-12.

15. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the method of any one of claims 1-12.

Images